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i«.  Abstract  This  report  is  the  second  in  the  investigation  of  the  roie  of  expert  cognitive 

processes  in  Aeronautical  Decision  Making  (ADM).  The  first  report  defined  the  differences 
between  expert  and  novice  decision  makers  and  correlated  the  development  of  expert  pilot 
cognitive  processes  with  training  and  experience.  This  volume  continues  the  research  into 
the  understanding  of  how  pilots  think  by  examining  human  information  processing  and 
expert  problem  solving  characteristics.  From  this  analysis:  the  mental  attributes  critical  to 
developing  expert  decision  making  are  explained,  and  three  basic  limits  on  decision  making 
are  identified.  These  are  attention  span,  short  term  memory  and  long  term  memory. 

Next,  the  analysis  examines  the  development  of  Expert  Decision  Making  (EDM)from  the 
comparison  of  how  novices  and  experts  perceive,  store,  organize  and  use  their  knowledge. 

This  analysis  shows  that  high  levels  of  competence  result  from  the  interaction  between 
knowledge  organization  and  processing  abilities.  Experts  are  found  to  possess  the  abilities 
of  rapid  access  to,  and  efficient  utilization  of  a  highly  organized  body  of  conceptual  and 
procedural  knowledge.  That  is,  an  elaborately  structured  set  of  associated  concepts, 
procedures  and  events  -  based  upon  many  years  of  study,  training  and  experience  in  a 
aviation.  The  use  of  this  knowledge  base  is  described  in  terms  of  the  characteristics 
of  EDM  related  to  perception,  attention,  memory,  creativity  and  superior  situational  awareness. 

Since  the  speed  and  accuracy  of  expert  cognitive  processes  make  them  take  on  the 
characteristics  of  insight  or  intuition,  the  role  of  intuition  in  decision  making  is  the  next 
analytical  part  of  the  investigation.  Intuition  is  defined  and  the  properties  of  intuition 
relevant  to  EDM  delineated. 

Finally,  the  report  concludes  by  presenting  alternative  EDM  training  opportunities. 

These  include:  training  to  enhance  memory,  computer  based  training  tools,  and  promising 
methods  for  testing  pilot*  which  will  also  provide  training  and  the  opportunity  to  learn 
the  cognitive  skills  of  experts  without  spending  all  of  the  learning  time  in  actual  flight. 
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PREFACE 


In  recent  years,  experienced  pilots  and  crews  have  adequately  responded  to  multiple 
failures  for  which  there  were  no  specified  procedures,  no  previous  simulator 
training  and  certainly  no  past  experience.  Explosive  decompressions  caused  by  cargo 
door  failures,  total  loss  of  triply  redundant  hydraulic  systems  and  fuselage  failures 
v  were  all  situations  which,  within  all  reasonable  statistical  criteria,  the  aircraft 

manufacturers,  the  airlines,  the  regulatory  agencies  and  the  insurance  companies 
could  not  conceive.  Yet,  these  pilots  and  crews  responded  quickly,  adapted  their 
trained  procedures  to  the  task  demands  of  the  crises  and  either  in  whole  or  in  part 
"saved  the  day". 

During  this  same  time  period,  first  generation  Aeronautical  Decision  Making 
(ADM)  training  was  developed  by  joint  efforts  of  the  FAA,  industry  and  academe. 

The  ADM  training,  in  some  cases  combined  with  Crew  Resource  Management 
training,  has  been  shown  to  significantly  reduce  the  percentage  of  accidents 
attributable  to  "human  error".  However,  this  initial  type  of  ADM  training,  a  serial, 
checklist  evaluation  of  alternative  decisions,  has  not  been  accepted  by  all  pilots  and 
is  not  readily  applied  in  "knowledge  lean",  time-pressured  multiple  failures. 
Consequently,  in  1990  the  FAA  initiated  an  effort  to  analyze  and  understand  the 
more  complex  decision  making  and  problem  solving  capabilities  of  the  high  time, 
experienced,  "expert"  pilot. 

This  is  the  second  report  in  the  effort  to  define  the  differences  between  novice  and 
expert  pilot  decision  making.  The  first  report,  "Introduction  to  Cognitive  Processes  of 
Expert  Pilots"  report  number  DOT/FAA/RD-92/12,  reviewed  accident  scenarios  and 
correlated  the  characteristics  of  these  pilot's  decisions  with  the  characteristics  of 
experts  in  other  fields.  It  also  analyzed  the  development  of  a  pilot's  knowledge  and 
type  of  information  processing  with  training  and  experience  to  show  that  many 
characteristics  of  Expert  Decision  Making  (EDM)  were  enhanced  as  pilots  accumulate 
flight  time.  The  current  report  is  an  attempt  to  explore  whether  or  not  the 
development  of  EDM  can  be  expedited  by  study  and  training  in  addition  to  the  basic 
requirement  for  actual  flight  experience. 
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EXECUTIVE  SUMMARY 


The  speed  and  accuracy  of  problem  solving  using  memory 
is  the  fundamental  process  that  accounts  for  differences 
between  expert  and  novice  pilot  thinking  capabilities. 

Sensing,  organizing  and  using  information  requires  the 
resources  of  short  term  and  long  term  memory  systems.  Both  types  of  memory  are  * 

critically  impacted  by  the  quantity  and  structure  of  the  available  information.  Since 
a  pilot's  stored  information  includes  aeronautical  knowledge  (facts),  procedural 
knowledge  (ATC,  aircraft  and  systems),  training,  attitudes,  emotions  and  general  * 

skills,  as  well  as  experience,  it  is  reasonable  to  expect  that  at  least  some  portion  of  the 
expert  thinking  capabilities  can  be  learned  without  relying  solely  on  knowledge 
gained  through  actual  flight  experience.  One  goal  of  this  research  is  to  identify  the 
expert  capabilities  that  could  be  learned  in  a  training  environment. 

Expert  decision  making  is  characterized  by  the  speed  and  accuracy  with  which 
information  can  be  retrieved  and  applied  to  new  situations.  The  type  of  knowledge 
available,  how  it  is  represented  (stored)  in  memory,  how  pilots  tie  it  together  and 
how  they  recall  (apply)  it  are  central  to  the  understanding  of  cognition  in  general 
and  how  expert  pilots  think  in  particular. 

This  study  first  describes  four  basic  steps  that  pilots  use  when  thinking  about  a 
situation  or  solving  a  problem.  Then,  the  performance  of  these  steps  is  related  to 
the  basic  human  information  processing  systems  used  for  problem  solving  and 
decision  making.  Finally,  the  unique  elements  of  those  systems  which  are  used 
most  efficiently  by  experts  are  described.  This  background  provides  the  knowledge 
and  understanding  needed  to  begin  development  of  expert  pilot  decision  making 
training  materials  which  is  the  second  goal  of  this  analysis.  A  brief  overview  of  the 
contents  and  findings  of  this  study  is  presented  in  this  section. 

The  focus  of  this  analysis  is  on  the  PREPARATION 
and  EXECUTION  aspects  of  how  expert  pilots  think 
since  these  two  areas  offer  the  greatest  potential  for 
improved  training.  For  this  analysis,  the 
PREPARATION  factor  was  limited  to  the  cognitive 
aspects  of  Sensing  and  Organizing  information. 

The  EXECUTION  factor  was  similarly  limited  to 
the  cognitive  aspects  of  Analyzing  and  Responding 
to  the  needs  of  the  situation.  Thus,  the  term 
Cognitive  SOARing  was  coined  to  recognize  the  * 

special  level  of  expertise  required  to  respond  to 
previously  inexperienced  or  abnormal  situations 
and  achieve  the  highest  level  of  "adaptive"  expert 
decision  making.  This  nomenclature  is  also 
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beneficial  to  establishing  the  ultimate  goal  for  Expert  Decision  Making  (EDM) 
training,  that  is,  adaptive  expertise.  However,  any  additional  training  in  the  mental 
characteristics  and  thinking  processes  described  for  cognitive  SOARing  will  be 
responsive  to  the  needs  of  the  industry  for  advanced  Aeronautical  Decision  Making 
(ADM)  training  which  generated  this  research. 

Sensing  is  the  first  preparation  step  involved  in  decision  making.  The  basic 
characteristic  of  sensory  memory  that  is  important  for  pilots  to  be  aware  of  is  that  a 
lot  of  information  is  "sensed"  or  received,  but  only  a  small  amount  is  "attended  to". 
Dedicated  time  spent  focusing  on  individual  cues  and  responding  is  time  taken 
away  from  situation  monitoring  or  passive  situation  assessment.  Attention 
training  can,  therefore,  provide  the  first  part  of  a  program  designed  to  lessen  the 
reliance  on  experience  as  the  only  means  of  attaining  expert  performance. 

Organizing  involves  filtering,  prioritizing  and  structuring  sensed  information. 
During  this  step,  short  term  and  long  term  memory  resources  are  used  to  identify 
the  most  important  information  and  develop  an  understanding  of  the  situation. 
Short  term  memory  and  long  term  memory  should  not  be  thought  of  as  different 
places  pilots  "put"  facts  or  procedures.  Rather,  the  differences  in  these  two  memory 
systems  are  based  upon  the  "operational  readiness"  of  the  knowledge  at  any  given 
time. 

SHORT  TERM  MEMORY  provides  active,  usable  chunks  of  information  in  a  state  of 
readiness  to  be  used.  The  precise  content,  organization  and  usability  of  each  chunk 
is  tied  directly  to  exposure  and  practice.  Therefore,  pilot  training  and  especially 
decision  making  training  could  benefit  by  re-examining  the  criticality  of 
information  chunking  and  training  novice  pilots  earlier  in  the  necessary  short  term 
memory  skills. 

LONG  TERM  MEMORY  provides  stored  information  including  factual,  procedural, 
experiential  and  emotional  knowledge.  Pilots  have  stored  this  knowledge  in  related 
patterns  or  schema  and  must  reactivate  it  based  upon  the  specific  situation. 
Reactivation  can  be  initiated  by  the  cues  (mental  or  physical),  the  context  of  a 
situation  (standard  procedure  or  emergency)  and  the  pilot's  abilities  to  make 
associations  between  current  and  previous  patterns. 

Analyzing  (or  information  processing  and  evaluation)  is  the  third  critical  step  in 
pilot  thinking.  This  step  relies  on  the  type  of  knowledge  stored  and  how  it  can  be 
retrieved.  Expert  pilots  have  developed  superior  long  term  memory  organizational 
capabilities  which  facilitate  RECOGNITION  and  RECALL.  Experts  use  schema, 
pattern  recognition,  associative  reasoning,  elaborations  and  inferences  to  interpret 
the  cues  and  context  of  a  new  situation  based  upon  their  related  knowledge. 

Responding  is  the  most  critical  step  of  the  Expert  Decision  Making  (EDM)  process. 
This  step  requires  that  the  pilot  take  some  action  to  alter  or  control  the  situation  and 
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then  monitor  the  effectiveness  of  that  action.  Responding  involves  the  use  of 
conceptual  and  procedural  knowledge.  Since  aviation  training  is  highly  procedural 
both  in  developing  flying  skills  (psychomotor)  and  in  problem  solving  for  normal 
and  emergency  situations,  pilot's  are  provided  the  foundation  for  more 
sophisticated  problem  solving  using  production  rules.  PRODUCTION  RULES 
consist  of  conceptual  knowledge  combined  with  general  problem  solving 
procedures  (i.e.,  rules-of-thumb,  working  backward  from  a  goal,  etc.)  to  create  new, 
problem  specific  procedures.  The  ability  to  develop  problem  specific  production 
rules  marks  the  early  beginnings  of  how  expert  pilots  think. 
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As  experience  is  gained  the  pilot  begins  to 
develop  associative  problem  solving  capabilities 
for  enhanced  decision  making.  As  a  result  of 
experience,  additional  flight  training  and 
possibly  a  knowledge  of  ADM  principles,  the 
pilot  develops  a  capacity  for  more  dynamic 
thinking.  Expert  pilots  can  create  new 
production  rules  and  patterns  in  the  vein  of 
"opportunistic  planning"  based  upon  unique, 
previously  unencountered  problem 
characteristics.  This  capability  to  creatively 
respond  to  unique  problems,  novel  task 
demands  and  chaotic  situations  identifies  the  highest  level  of  EDM.  At  this  level, 
the  pilot  has  become  an  "adaptive"  expert.  The  coordinated  use  of  cues  and  context 
with  stored  schema  (i.e.,  perceptual  superiority)  is  believed  to  be  a  "major 
mechanism"  used  by  experts  to  infer  unobserved  or  unknown  elements  of  a 
problem  in  knowledge  lean  or  chaotic  situations  (Anderson  1985). 
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The  expert  pilot's  perception  of  the  whole  situation  involves  mental 
representations  that  include  physical,  cognitive  and  internal  effects  which  are  used 
to  both  store  and  retrieve  knowledge.  Long  familiarization  in  a  specific  field  of 
knowledge  transforms  the  experts  mental  representations  into  an  accessible  form  of 
global  knowledge.  When  applied  to  new  problems  or  working  situations  the  expert 
decision  making  capabilities  occur  so  fast  that  they  appear  to  take  on  the 
characteristics  of  instantaneous  insight  or  intuition. 


Operationally,  INTUITION  is  critically  important  to  EDM.  The  most  meaningful 
definition  of  intuition  from  a  training  viewpoint  is:  "knowledge  based  on 
experiences  and  acquired  through  sensory  contact"  (Bastic,  1982).  In  this  context, 
intuition  can  be  viewed  as  an  inferential  process  similar  to  ordinary  perceptions. 
That  is,  intuition  does  not  advance  in  careful,  well-planned  steps.  It  tends  to 
involve  responding  to  situations  based  seemingly  on  an  implicit  perception  of  the 
whole  problem.  As  with  other  procedural  knowledge,  the  pilot  arrives  at  an  answer 
and  acts  on  it  with  little  or  no  awareness  of  the  process  by  which  it  was  reached. 

This  knowledge  cannot  be  verbalized,  but  is  activated  by  the  cues  and  context  of  the 
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situation.  Nevertheless,  intuition  is  a  highly  rational  skill  and  one  that  is 
sometimes  necessary  for  pilots  to  use.  Intuition  is  simply  an  extension  of  the  logical 
problem  solving  process  and  an  integral  part  of  the  "autonomous"  stage  of  how 
pilots  think.  This  stage  requires  extensive  knowledge  of  the  subject  matter  and  basic 
principles  in  aeronautics  and  extensive  experience  in  aviation.  It  is  the  fast  access  to 
large  meaningful  patterns  of  knowledge  that  "appears"  intuitive  to  the  observer 


The  primary  focus  of  this  analysis  was  on  the  capabilities  and 
limits  of  the  pilots  sensory,  short  and  long  term  memory  systems 
since  memory  is  the  fundamental  process  that  accounts  for 
%  differences  in  expert  and  novice  pilot  thinking  capabilities  . 

Enhanced  memory  training  to  develop  EDM  in  pilots  is  discussed  in  Sections  3.4 
and  5.2.  Secondly,  the  pilots  Problem  Solving  abilities  were  related  to  type  of 
knowledge,  its  organization  and  the  process  differences  between  novices  and 
experts.  The  purpose  of  this  discussion  was  to  expand  the  understanding  of  how  the 
SOARing  processes  were  accomplished  at  the  cognitive  level  and  to  specifically 
identify  factors  which  could  offer  opportunities  for  enhanced  training.  These 
opportunities  are  discussed  in  Sections  4.1  and  5.4. 
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1.  ATTENTION 

2.  PERCEPTION 

3.  CHUNKING 

4.  SCHEMA 

5.  PATTERN  RECOGNITION 

6.  CUES  &  CONTEXT 

7.  RECALL 

8.  ASSOCIATIONS 

9.  ELABORATIONS 

10.  INFERENCES 

11.  PROCEDURAL  KNOWLEDGE 

12.  PRODUCTION  RULES 


INFORMATION 
PROCESSING  SYSTEM 

SENSORY  MEMORY 
SENSORY  MEMORY 
SHORT  TERM  MEMORY 
LONG  TERM  MEMORY 
LONGTERM  MEMORY 
LONGTERM  MEMORY 
LONGTERM  MEMORY 
LONGTERM  MEMORY 
LONGTERM  MEMORY 
LONG  TERM  MEMORY 
PROCESSOR 
PROCESSOR 


Cognitive  SOARing  Attributes:  In 
total,  the  12  mental  attributes  listed  on 
the  left  were  identified  as  offering 
potential  for  developing  future  Expert 
Decision  Making  (EDM)  training 
methods  and  materials.  Other  fields 
offer  existing  training  programs  for 
enhancing  these  mental  attributes. 
These  can  be  used  as  resources  to 
expedite  the  development  of  EDM 
training  programs  for  pilots.  These  are 
addressed  and  their  adaptation 
explored  in  Section  5.2  and  App.  D. 


Expert  Decision  Making  Attributes:  There  are  seven  basic  characteristics  of  Expert 
Decision  Making  which  are  extremely  relevant  to  the  desire  to  develop  EDM 
training  materials  for  pilots.  These  distinguishing  characteristics  have  been 
identified  in  many  fields  of  expertise,  but  they  have  particular  importance  in  the 
current  analysis  of  how  pilots  use  their  knowledge,  training  and  experience  to 
overcome  the  adversities  associated  with  everyday  flight,  i.e.,  how  pilots  think. 
They  are  summarized  for  ease  of  reference  as  follows: 
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1 .  Superior  Short  Term  and  Long  Term 
Memory 


Goal  Orien;ed 


3.  Fast  Access 


4.  Opportunistic  Planning 


5.  Adaptive 


6.  Self-Monitoring 


7.  Perceptual  Superiority 


Experts’  recall  seems  to  exceed  the  limits  of  Short  Term  Memory. 
However,  it  is  the  autonomous  information  processing  of  many  of  their 
skills  which  frees-up  greater  storage.  Elaborations,  associations  and 
inferences  expedite  Long  Term  recall  as  well. 

The  knowledge  of  experts  is  highly  procedural  and  goal  oriented.  Concepts 
are  bound  to  procedures  for  their  applications  and  to  conditions  (contexts) 
under  which  these  procedures  are  useful. 

Experts  can  solve  problems  quickly  and  accurately  for  three  reasons.  They 
are  faster  at  the  skill  based  tasks.  The  faster  skill  performance  frees-up 
working  memory  for  processing  other  aspects  of  the  problem.  They  can 
arrive  at  a  solution  without  conducting  extensive  search  of  memory. 

Experts  develop  the  capability  to  revise  production  rules  and  to 
simultaneously  access  multiple  possible  interpretations  of  a  situation.  The 
development  of  this  capability  is  influenced  by  task  demands  and 
experience. 

There  are  both  routine  and  adaptive  experts.  Either  type  is  outstanding  in 
terms  of  speed,  accuracy  and  automaticity  of  their  decision  making. 
Adaptive  experts  possess  the  additional  ability  to  creatively  respond  and 
develop  solutions  to  knowledge  lean  or  ambiguous  situations. 

The  greater  knowledge  bases  and  different  knowledge  representations  of 
experts  allow  more  time  to  predict  problem  difficulty  on  the  basis  of 
underlying  principles  and  more  time  to  monitor  accurately  how  they  should 
allocate  their  time  for  solving  problems. 

Experts  have  the  ability  to  perceive  large  meaningful  patterns  due  to  the 
organization  of  their  knowledge  base.  This  organization  is  based  on 
experience  and  training.  The  pattern  recognition  and  recall  occurs  so  rapidly 
that  it  takes  on  the  characteristics  of  instantaneous  insight  or  intuition. 


In  summary,  we  have  discussed  the 
~  importance  of  the  cues  and  context  of 

f  the  environment  or  the  stimuli  which 

"trigger"  EDM  or  adaptive  problem 

'srCT*'  1 — solving  behavior.  The  importance  of 

;oET™Ei  I — I— i  attentional  resources,  perception  and 

Hi  ►  /  I  a  enow  r  r 

ESSE?""]  Nni?f  — f  motor  skills  in  "aiding  and  abetting" 

-  EDM  have  been  identified.  The  roles 

-K^phocessor^ - tip****  \  0f  short  term  and  long  term  memory, 

ESSm  ZT  I  including  chunking  and  schema,  have 

>!Eem  **  sSL  also  been  shown  as  critical  to  the  pilot's 

v>s** - strategic  mobility  and  intuition. 

Finally  and  most  importantly  from  a 

training  perspective,  the  foundation  of  the  pilot  knowledge  and  its  organization 
have  been  identified  as  critical.  These  areas,  attention,  perception,  memory,  type  of 
knowledge  and  its  organization  could  be  used  as  the  crystal  about  which  EDM 
training  programs,  materials,  tools  and  methods  can  grow. 
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Expert  pilots  need  to  know  how  to  use  the  full  range  of  their  "thinking"'  capabilities 
depending  on  the  cues,  context,  familiarity  and  difficulty  of  the  situation.  Research 
in  many  fields  has  shown  that  at  least  some  of  the  expert  decision  making  skills  can 
be  acquired  through  practice  and  training.  The  ultimate  purpose  of  this  research  is 
to  suggest  a  starting  point  for  developing  EDM  training  for  pilots.  The  following 
conclusions  and  recommendations  are  tailored  to  that  purpose. 

CONCLUSION  1:  Enhanced  memory  and  problem  solving  training  are  required  to 
expedite  development  of  Expert  Decision  Making  (EDM)  in  novice  pilots. 

Recommendation  —  Aviation  problem  solving  and  pilot  memory  training  courses 
should  be  developed.  Three  non-aviation  cognitive  training  programs  which  offer 
potential  insight  for  new  EDM  training  paradigms  were  reviewed  in  Section  5.2. 
These  and  other  programs  which  have  documented  improvements  in  attention, 
memory  and  problem  solving  should  be  explored  and  adapted  to  prototype  EDM 
courseware. 

CONCLUSION  2:  The  reliance  of  EDM  on  knowledge  based  on  experiences  and 
sensory  stimuli  make  the  selection  of  training  media  extremely  critical. 

Recommendation  —  The  development  of  a  knowledge  based  CBT  or  IVT  expert 
training  device  should  be  initiated.  The  fields  of  Activity  Based  Learning  (ABL), 
Computer  Based  Training  (CBT)  and  Interactive  Video  Training  (IVT)  offer  near 
term,  cost  effective  alternatives  which  are  attractive  to  both  air  lines  and  general 
aviation  pilots.  Two  example  CBT  systems  were  discussed  in  Section  5.2. 

CONCLUSION  3:  Experience  since  1986  with  ADM  training  in  both  civil  and 
military  U.  S.  operations  has  shown  that  decision  making  can  be  taught  and  this 
training  reduces  the  number  of  human  error  accidents. 

Recommendation  --  A  second  generation  ADM  or  EDM  training  manual  should  be 
developed  in  response  to  industry's  request  for  additional  material  and  to  continue 
the  safety  improvements  generated  by  this  training.  Much  of  the  material  needed 
for  this  manual  has  already  been  created  during  Phases  I  and  II  of  this  project. 
Reformatting  the  EDM  research  into  a  training  manual  could  be  accomplished  in  a 
timely,  cost  effective  manner. 
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CONCLUSION  4:  An  FAA  EDM  training  program  should  be  developed  and  tested. 
Courseware,  an  instructor's  syllabus,  student  manuals  and  self  test  materials  are 
required. 


Recommendation  --  The  aviation  EDM  training/testing  conducted  at  Massey 
University  in  New  Zealand  (Section  5.4)  since  1990  should  be  the  starting  point  for 
an  FAA  EDM  course  for  pilots.  The  Intuitive  Decision  Making  testing  from  non¬ 
aviation  fields  (Section  5.4)  should  be  reviewed  for  possible  relevance  or 
enhancements  of  the  Massey  program. 
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CONCLUSION  5:  A  written  self- 
assessment  test  would  be  the  most 
expeditious  and  cost  effective  method  of 
raising  the  awareness  of  aviator  EDM 
characteristics  and  initiating  EDM 
training. 

Recommendation  --  Continue 
development  of  the  Cognitive  SOARing 
self-assessment  test  initiated  in  Section 
5.4  for  rating  individual  pilot  EDM 
capabilities. 


CONCLUSION  6:  Selecting  realistic  EDM  scenarios  with  critical  events  and 
performing  a  cognitive  task  analysis  of  novice  vs.  expert  decision  making  for  these 
events  is  a  basic  requirement  for  future  EDM  training  development. 


Recommendation  —  The  design  of  EDM  training  assessment  methods  should  be 
initiated  in  parallel  with  training  materials,  training  tools  and  courseware 
development.  A  methodology  and  implications  of  developing  EDM  scenarios  are 
presented  in  Appendix  F.  This  type  of  methodology  is  required  for  Activity  Based 
Learning  (Recommendation  2),  a  second  generation  training  manual 
(Recommendation  3)  d  self-assessment  written  testing  (Recommendation  5). 
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Due  to  the  number  of  topics,  the  complex  interrelationships,  and  the  desire  to  make 
this  report  as  easy  to  read  as  possible,  the  following  diagram  is  provided  as  a  guide  or 
road  map.  The  boxes  indicate  the  type  and  sequence  of  the  primary  subject  matter 
discussed.  As  you  proceed  from  section-to-section,  refer  back  to  this  road  map  to 
retain  the  perspective  of  where  you  are  in  the  discussion  and  how  the  material 
relates  to  the  overall  EDM  process. 


TRAINING  APPROACHES 


HOW  EXPERT  PILOTS  THINK 

Cognitive  Processes  in  Expert  Decision  Making 


1.0  PURPOSE 

Both  operators  and  researchers  have  recognized  decision  making  --  the  ability  of  a 
pilot  to  respond  to  cues  from  the  environment,  evaluate  the  situation,  come  to 
conclusions  and  act  on  those  conclusions  —  as  possibly  the  only  reason  for  keeping 
a  human  on  the  flight  deck  in  the  future  (Green  &  Muir,  1991).  This  research  is  an 
attempt  to  articulate  steps  for  development  of  training  materials  and  formulation  of 
training  tools  responsive  to  the  need  for  early  development  of  these  skills. 

The  ability  to  identify  a  problem  while  it  develops,  often  before  it  has  fully 
developed,  select  an  optimal  solution  and  then  implement  it  at  the  right  time  is  the 
distinguishing  trait  of  an  expert  pilot.  This  ability  to  anticipate  trouble  and  respond 
in  a  timely  manner,  with  confidence,  involves  both  natural  and  learned  behavior 
employing  perception,  memory,  problem  solving  and  decision  making  skills. 
Expert's  use  forward  thinking  and  start  solving  problems  from  an  intuitive 
perception  of  the  situation.  These  perceptions  are  based  upon  the  context  and  cues 
available  from  the  situation  as  well  as  experience  and  training.  The  latest  research 
has  shown  that  these  abilities  can  be  learned  through  tailored  training  (Agor,  1986, 
Firestein  &  McCowan,  1988  and  Hogarth  1989).  This  report  provides  the  background 
of  primary  decision  making  characteristics  of  experts  in  order  to  establish,  a  common 
foundation  of  knowledge  for  addressing  how  pilots  can  be  trained  expert  decision 
making. 

The  objectives  of  this  study  are: 

•  To  identify  the  expert  decision  making  characteristics  that  could  be  learned  in  a 
training  environment,  and 

•  To  provide  the  knowledge  and  understanding  needed  to  begin  development  of 
expert  pilot  decision  making  training  materials  and  tools. 

This  will  require  examining  the  mental  processes  used  by  expert  pilots  and  the 
characteristic  thinking  abilities  they  have  developed  to  meet  the  demands  of  their 
flying  tasks.  Specifically,  the  processes  and  abilities  needed  by  pilots  to  perform  fast, 
accurate  decision  making  in  normal  situations,  trained  emergency  situations  and 
abnormal  emergencies  will  be  analyzed. 


2.0  BACKGROUND:  ADM  Training  History,  Successes  and  Limitations 

The  work  described  in  this  report  was  initiated  in  October  of  1990  with  a  Phase  I 
study  which  investigated  the  differences  between  novice  and  expert  pilot  decision 
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making  from  an  information  processing  perspective  (Adams  &  Ericsson,  1992).  The 
Phase  I  report  correlated  the  development  of  expert  decision  making  with  pilot 
training  and  experience,  and  reviewed  accident  scenarios  which  exemplified  those 
processes.  The  need  for  this  introductory  study  evolved  from  10  years  of 
Aeronautical  Decision  Making  (ADM)  research  (1975-1985)  sponsored  primarily  by 
the  Federal  Aviation  Administration  and  performed  by  industry  and  academia. 

This  research  lead  to  the  publication  of  a  series  of  10  training  manuals  and  reports 
on  ADM  (1986-1988).  Topics  included  the  range  of  airplane  pilot  ADM  training 
manuals  from  student/private  candidates  to  multicrew  resource  management,  as 
well  as,  reports  devoted  to  helicopter  pilots,  flight  instructors  and  EMS. 

Extensive  experimental  validations  and  empirical  testing 
in  both  civil  and  military  operational  environments 
have  documented  that  accident  rate  reductions  of  about 
50%  can  be  seen  when  comparing  pilot  groups  with  and 
without  decision  making  training.  Although  it  is 
difficult  to  accurately  assess  the  impact  of  these  manuals 
throughout  aviation,  significant  reductions  in  Human 
Error  (HE)  accidents  have  been  demonstrated  in  the 
specific  aviation  applications  shown  at  the  left  (Diehl, 
1991). 
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This  basic  research  defined  judgment  as:  the  ability  to  stay  on  top  of  and  control  the 
flight  situation,  and  the  motivation  to  assure  safety  through  timely  decisions.  ADM 
identified  hazardous  attitudes  in  flight  operations  and  provided  pilots  with  a  self- 
assessment  test  by  which  to  judge  their  own  abilities.  The  method  stressed 

situational  awareness  and  a  structured  approach  to 
decision  making  to  enhance  the  pilot's  application  of 
conventional  flight  training,  knowledge,  skill  and 
experience.  The  methods  taught  to  accomplish  good 
decision  making  stressed  serial,  deductive  reasoning  in 
a  checklist  form  using  the  DECIDE  model  (Detect, 
Estimate,  Choose,  Identify,  Do,  Evaluate).  This  method 
is  useful  to  notices,  but  not  necessarily  representative 
of  the  more  advanced  decision  making  abilities  used  by 
expert  pilots. 


BASIC 

AERONAUTICAL  DECISION  MAKING 
(ADM) 

•  Mined  Judgment 

•  IdanlNIed  hazardous  attitudes 

•  Developed  eed-tast 

•  Streeaod  sMuat  tonal  awareness 

•  Utlltzad  the  "Decide*  Model 


The  specific  shortcomings  of  this  approach  included:  the  great  difficulty  in  carrying 
out  the  linear  analysis  under  conditions  of  time  pressure,  the  difficulty  in  applying 
it  to  problems  with  incomplete  information  or  ambiguous  data,  and  that  it  was  not 
representative  of  documented  differences  between  novice  and  expert  decision 
makers  in  other  fields  (Chi  &  Glaser,  1988). 
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All  of  these  shortcomings  lead  the  industry  to  come  back  to  the  FAA  with  a  request 
for  additional  training  material  for  use  in  training  of  novice  pilots  and  for  use  in 
both  initial  and  recurrency  training  with  more  experienced  pilots. 


3.0  EXPERT  DECISION  MAKING  (EDM) 

Developing  EDM  training  involves  unraveling  the  relationships  between  cognition 
(how  pilot's  think  in  operational  situations)  and  training.  Conventional  pilot 
training  has  been  based  upon  a  foundation  of  skill  based,  rule  based  and  knowledge 
based  tasks.  That  is,  pilots  are  taught  conceptual  knowledge;  flying  procedures;  and, 
basic  pilot  skills  while  leaving  the  development  of  decision  making  to  the  realm  of 
experience.  The  novice  pilot,  therefore,  is  expected  to  learn:  aerodynamics,  airplane 
performance,  electrical  and  hydraulic  systems,  Federal  Aviation  Regulations,  etc. 

He  is  then  trained  in  aircraft  control  and  operation  for  both  normal  and  emergency 
situations.  This  training  includes  procedures  development  for  preflight,  takeoff, 
cruise,  approach  and  landing  phases  of  flight.  Through  this  training,  the  novice 
develops  and  improves  his  basic  psychomotor  abilities  and  hones  his  flying  skills. 

At  this  stage,  decision  making  is  only  taught  informally  through  training  session 
debriefs,  hangar  flying,  analyses  of  other  pilots  experiences  and  the  limited  flight 
experience  gained  in  preparation  for  an  airman  certification  test.  After  successfully 
passing  the  test,  the  novice  pilot  is  expected  to  cautiously  begin  developing  good 
decision  making  and  judgment  skills  as  he  gains  experience.  Because  of  the 
emphasis  of  aviation  on  procedure  oriented  training  both  in  developing  flying  skills 
and  in  decision  making  skills,  training  lays  the  foundation  for  the  development  of 
more  sophisticated  decision  making  as  experience  is  gained. 

3.1  EXPERTISE  IN  ACTION:  The  Type  C  Pilot 

Common  characteristics  of  expert  decision  making  have  been  observed  in  the  fields 
of  mathematics,  physics,  medicine,  music,  sports  and  aviation  (Ericsson,  1991).  In 
fact,  individuals  with  expert  thinking  ability  have  been  identified  as  a  new 
personality  type.  First  the  classic  Type  A  and  B  personalities  were  identified, 
individuals  who  respond  dramatically  differently  to  stressful  situations.  Then  the 
Type  T  or  Thrill-seeking  personalities,  those  who  seek  out  the  "edge-of-the- 
envelope"  and  enjoy  the  challenge  of  overcoming  dangerous  situations  was 
recognized.  Now  psychologists  have  identified  the  Type  C  personality  style  as  Chaos 
changing  individuals  who  are  expert  problem  solvers.  Type  C  individuals  have  a 
tolerance  for  ambiguity,  can  see  solutions  in  unfamiliar  and  information  lean 
environments  and  develop  action  plans  even  in  time  compressed  situations 
(Buffington,  1989). 

The  most  dramatic  examples  of  how  Type  C  pilots  and  crews  apply  their  expert 
thinking  skills  have  occurred  in  several  airline  accidents  listed  on  the  following 
page.  The  catastrophic  engine  failure  and  subsequent  total  failure  of  the  triply 
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HEROIN AUTIl'OAlL  DEC3S30M  redundant  flight  control  system  of  a 
(HMKMNIQ  United  Air  Lines  DC-10  (Flight  232), 

the  failure  of  a  cargo  door  and 

EXPERTISE  IN  ACTION  explosive  decompression  of  another 

United  B-747  (Flight  811),  and  the 
fuselage  separation  of  an  Aloha  Air 
Lines  B-737  (Flight  243)  are  three 
prime  examples  of  abnormal 
situations  in  which  Type  C  problem 
solving  abilities  were  used  to  overcome  chaotic  situations  for  which  there  were  no 
specified  procedures,  no  previous  simulator  training  and  certainly  no  past 
experience.  The  event  histories  of  these  accidents  were  analyzed  during  Phase  I  of 
this  project  to  provide  specific  examples  of  how  expert  pilots  think  (Adams  & 
Ericsson,  1992).  For  ease  of  reference,  the  expert  pilot  decision  making  process 
demonstrated  by  Captain  A1  Haynes  of  United  Flight  232  will  be  reviewed  here. 
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In  a  speech  on  January  26,  1991,  Captain  Haynes  reported  that  the  transition  from  a 
normal,  uneventful  takeoff  and  climb  to  37,000  feet  to  a  "nearly  uncontrollable" 

aircraft  occurred  in  about  15  seconds. 


EXPERTISE  IN  ACTION 
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His  first  response  was  reverting  to 
basic  airmanship  skills  (i.e.,  figuring 
out  how  to  fly  the  airplane).  His 
ingrained  training  in  crew  resource 
management  was  evident  by:  his 
immediate  decision  to  use  the  abilities 
of  a  check  airman  to  operate  the 
throttles  and  maintain  heading  with 
differential  thrust;  his  utilization  of 
the  second  officer  for  damage 
assessment;  his  concern  for  the 
passengers  and  coordination  with  the 
flight  attendant;  and,  his  professional 
communication  with  the  air  traffic 
controller. 


Captain  Haynes'  behavior  clearly  show  the  expert  pilots  ability  to  assimilate  data  and 
impressions  quickly,  formulate  a  solution  and  carry  it  out  while  maintaining 
mental  composure  under  extreme  time  pressures.  However,  even  with  complete 
utilization  of  his  expertise  and  all  available  personnel,  pitch  oscillations  (60  second 
phugoids)  and  roll  reversals  (from  4-28  degrees  of  bank)  were  as  stable  an  approach 
as  the  aircraft  could  make.  Regardless,  Captain  Haynes  reported  that  he  was  always 
"confident  of  getting  the  aircraft  on  the  ground".  This  expert  thinking  —  knowing 
what  to  do  and  when  to  do  it  —  and  the  taming  of  a  chaotic  situation  is  the  real  mark 
of  the  Type  C  decision  maker. 
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After  the  accident,  Captain  Haynes  stated  that  the  five  factors  listed  on  the  left  were 
instrumental  in  his  ability  to  land  the  aircraft  and  save  as  many  lives  as  he  did.  The 

focus  of  this  report  is  on  the  PREPARATION  and 
EXECUTION  aspects  of  how  expert  pilots  think.  Since  the 
way  these  tasks  are  performed  in  practice  strongly  affects 
how  they  will  be  performed  during  an  emergency,  these 
areas  offer  the  greatest  potential  for  improved  training. 
The  factors  and  processes  involved  in  these  two  critical 
decision  making  areas  include:  Sensing,  Organizing, 
Analyzing  and  Responding  to  the  cues  and  contexts  of  the 
situation.  The  Type  C  behavior  documented  by  Captain 
Haynes,  his  crew  and  the  crews  of  the  other  "aviation 
saves"  is  referred  to  herein  as  Cognitive  SOARing  to 
recognize  the  special  level  of  expertise  required  for  abnormal  emergencies.  This 
nomenclature  is  also  beneficial  to  establishing  an  ultimate  goal  for  Expert  Decision 
Making  (EDM)  training.  However,  any  additional  training  in  the  mental 
characteristics  and  thinking  processes  described  for  Cognitive  Soaring  will  be 
responsive  to  the  needs  of  the  industry  which  generated  this  research  project. 

3.2  COGNI  TIVE  SOARING:  Sensing,  Organizing,  Analyzing  and  Responding 

The  study  of  cognition  over  the  past  40  years  has  identified  the  importance  of  four 

systems  used  in  thinking  or  "information 
processing".  These  systems  are  illustrated  in  the 
Human  Information  Processing  (HIP)  diagram. 
Basically,  the  human  information  processing  system 
includes:  the  sensory  systems  (visual,  auditory,  seat- 
c-f-the-pants,  etc.),  the  memory  systems  (long  term, 
short  term  and  sensory  memory),  the  processor  and 
the  response  systems  (motor  events, 
communications,  etc.).  These  four  systems 
incorporate  the  basic  characteristic  concepts  which 
contribute  to  the  individual's  cognitive  SOARing  capabilities.  Equally  important  to 
the  expert  pilot's  thinking  are  the  types  of  knowledge  and  types  of  processing 
available  for  use  with  these  systems.  These  factors  as  well  as  the  characteristic 
concepts  will  provide  the  focus  for  the  following  description.  A  more  detailed 
examination  of  important  memory  and  problem  solving  characteristics  is  provided 
in  Appendix  A. 

Sensing  is  the  first  preparation  step  involved  in  decision  making.  Pilots  vary  in  the 
way  they  perceive  (recognize  and  sort)  information  from  the  cues  and  context  of  a 
situation.  SENSORY  MEMORY  provides  enough  retention  to  allow  a  reasoned 
response  to  each  situation.  The  basic  characteristic  of  sensory  memory  that  is 
important  for  pilots  to  be  aware  of  is  that  a  lot  of  information  is  "sensed"  or 
received,  but  only  a  small  amount  is  "attended  to".  Dedicated  time  spent  focusing 
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on  individual  cues  and  responding  is  time  taken  away  from  situation  monitoring  or 
passive  situation  assessment. 

Since  the  amount  of  time  pilots  spend  actively  attending  to  sensory  inputs  vs. 
passively  monitoring  the  cues  and  context  of  a  situation  varies  directly  with 
knowledge,  training,  experience  and  currency,  each  of  these  elements  impact  the 
pilots  ability  to  respond  in  a  timely  fashion  to  specific  situations.  Consequently, 
ATTENTION  is  one  of  the  differentiators  which  can  be  used  to  identify  experts  vs. 
novices.  Attention  training  can,  therefore,  provide  the  first  part  of  a  program 
designed  to  lessen  the  reliance  on  experience  as  the  only  means  of  attaining  expert 
performance. 

Organizing  involves  filtering,  prioritizing  and  structuring  sensed  information. 
During  this  step,  short  term  and  long  term  memory  resources  are  used  to  identify 
the  most  important  information  and  develop  an  understanding  of  the  situation  or 
problem.  This  understanding  is  formulated  into  a  group  of  related  facts,  data, 
results  and  procedures,  that  is,  a  pattern  which  characterizes  the  current  situation 
and  can  be  used  to  retrieve  related  information  from  short  and  long  term  memory. 
Although  the  novice  and  expert  pilot  have  equal  capability  for  cognitive  processing, 
novices  typically  use  lots  of  search  and  processing  time  in  a  less  focused  and  more 
general  manner.  The  outstanding  performance  of  experts  is  derived  from  how  their 
knowldege  is  structured  in  short  and  long  term  memory  for  retrieval,  pattern 
recognition  and  inference. 

Short  term  memory  and  long  term  memory  should  not  be  thought  of  as  different 
places  pilots  "put"  facts  or  procedures.  Rather,  the  differences  in  these  two  memory 
systems  are  based  upon  the  "operational  readiness"  of  the  knowledge  at  any  given 
time.  SHORT  TERM  MEMORY  provides  active,  usable  chunks  of  information  in  a 
state  of  readiness  to  be  used.  T  he  precise  content,  organization  and  usability  of  each 
chunk  is  tied  directly  to  exposure  and  pracitce.  Therefore,  pilot  training  and 
especially  decision  making  training  could  benefit  by  re-examining  the  criticality  of 
information  chunking  and  train  novice  pilots  earlier  in  the  necessary  short  term 
memory  skills. 

LONG  TERM  MEMORY  provides  stored  information  including  factual,  procedural, 
experiential  and  emotional  knowledge.  Pilots  have  strored  this  knowledge  in 
related  groups  or  schema  and  must  reactivate  it  based  upon  the  specific  situation. 
Reactivation  can  be  initiated  by  the  cues  (mental  or  physical),  the  context  of  a 
situation  (standard  procedure  or  emergency)  and  the  pilot's  abilities  to  make 
associations  between  current  and  previous  patterns.  Long  term  memory,  then, 
depends  on  the  pilot's  ability  to  respond  to  new  demands  for  information  through 
his  abilities  of  recognition  and  recall. 

Analyzing  (or  information  processing  and  evaluation)  is  the  third  critical  step  in 
pilot  thinking.  This  step  relies  on  the  type  of  knowledge  stored  and  how  it  can  be 
retrieved.  Once  again,  expert  pilots  have  developed  superior  long  term  memory 
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organizational  capabilities  which  facilitate  RECOGNITION  and  RECALL.  Experts 
use  schema,  pattern  recognition,  associative  reasoning,  elaborations  and  inferences 
to  interpret  the  cues  and  context  of  a  new  situation  based  upon  their  related 
knowledge.  This  expert  capability  manifests  itself  in  the  ability  to  intuitively 
respond  to  patterns  without  decomposing  them  into  component  features  or 
problem  elements.  An  example  of  these  abilities  would  be  the  expert's  ability  to 
respond  to  loss  of  an  engine  on  takeoff  without  consciously  "thinking  through"  the 
engine  out  procedure.  This  understanding  occurs  effortlessly  due  to  the  experts 
knowledge  structure. 

The  experts  ability  to  fast  access  their  schema  (concepts  or  patterns)  of  aviation 
knowledge  is  expedited  by  the  ASSOCIATIONS  with  cues  and  context  of  new 
situations  which  stimulate  the  recall  process.  Although  the  associations  (or  concept 

ELABORATIONS)  are  predominantly  based  upon 
experience  today,  training  aimed  at  replicating  this 
ability  is  not  an  unreasonable  goal.  In  addition  to  the 
H]p ..  PROBLEM  SOLVING  elaborations,  the  expert's  ability  to  use  INFERENCES 

to  aid  reconstruction  of  similar  problems  and 
- solutions  could  also  comprise  part  of  this  training. 

CCMCZPDWU.  >  "FACTS  ft  THINGS" 

wwciinuRAL  « "cocNtTivE ActivmES"  Responding  is  the  most  critical  step  of  the  Expert 

Decision  Making  (EDM)  process.  This  step  requires 
that  the  pilot  take  some  action  to  alter  or  control  the 
situation  and  then  monitor  the  effectiveness  of  that 
action.  Responding  involves  the  use  of  conceptual 
and  procedural  knowledge.  Since  aviation  training 
is  highly  procedural  both  in  developing  flying  skills  (psychomotor)  and  in  problem 
solving  for  normal  and  emergency  situations,  pilot's  are  provided  the  foundation 
for  more  sophisticated  problem  solving  using  production  rules.  PRODUCTION 
RULES  consist  of  conceptual  knowledge  combined  with  general  problem  solving 
procedures  (i.e.,  heuristics,  algorithms,  working  backward  from  a  goal,  etc.)  to  create 
new,  problem  specific  procedures.  How  the  application  of  problem  specific 
production  rules  develops  in  pilots  is  summarized  in  Section  3.3.  This  ability  marks 
the  early  begininnings  of  how  expert  pilots  think.  As  these  rules  are  used  more  and 
more  often,  and  applied  to  many  situations,  they  result  in  autonomous  generation 
of  specilized  production  rules  which  often  use  forward  inferencing  to  progress  from 
the  initial  problem  toward  a  solution  or  goal. 

3.3  PROBLEM  SOLVING:  Expert  vs.  Novice 

Problem  solving  is  defined  as  behavior  that  is  task  dependent  and  purposeful  (goal 
oriented).  Therefore,  the  following  discussion  is  about  how  pilots  use  their 
knowledge,  thinking  skills  and  the  cues  or  stimuli  of  a  situation  to  develop  problem 
solving  approaches.  These  very  often  cannot  be  completely  or  adequately 
verbalized;  for  example,  how  to  properly  judge  and  consistently  perform  the  landing 
flare  could  be  broken  down  and  described  but  the  result  would  lose  the  essentials  of 
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the  "how-to".  To  begin  to  understand  the  problem  solving  process,  it  is  necessary  to 
examine  some  basic  differences  between  how  novices  and  experts  approach  problem 
solving. 

Problem  solving  can  be  described  in  terms  of  situational  awareness  and  decision 
making  in  aviation.  A  pilot  is  expected  to  do  more  than  skillfully  resolve 
emergencies  when  they  occur.  It  is  equally  important  to  actively  avoid  situations 
that  might  lead  to  emergencies  by  sensing  and  identifying  early  signs  of  impending 
problems  and  taking  corrective  action  before  a  critical  situation  develops.  This 
responsibility  requires  vigilant  monitoring  of  the  environment  and  the  situation 
relative  to  the  planned  sequence  of  events  during  a  flight.  Problem  solving  can  be 
thought  of  as  a  search  for  ways  to  return  to  the  desired  plan  or  the  setting  of  a  new 
goal  once  a  change  or  deviation  from  the  desired  plan  is  recognized. 

This  section  is  about  the  knowledge  underlying  novice  and  expert  problem  solving 
activities.  It  presents  a  major  change  of  emphasis  in  the  analysis  of  how  pilots 
think.  Up  to  this  point,  the  analysis  has  been  concerned  with  how  knowledge  about 
the  environment  or  situation  gets  into  the  system  (i.e.,  is  sensed  and  perceived)  and 
how  it  is  stored  (chunks  and  schema)  in,  and  retrieved  (recall  and  recognition)  from, 
memory.  This  kind  of  knowledge  is  referred  to  as  declarative  knowledge  or 
conceptual  knowledge.  Conceptual  knowledge  consists  of  knowledge  that  can  be 
verbalized  or  knowledge  about  "facts  and  things".  Problem  solving  uses  this  type  of 
knowledge  as  a  basic  resource,  but  requires  another  type  of  knowledge.  Procedural 
knowledge,  to  understand  how  sensed  and  perceived  information  is  used  in  the 
thinking  process. 

Procedural  knowledge  is  knowledge  about  actions  or  how  to  perform  various 
cognitive  activities.  This  is  the  type  of  knowledge  you  learn  by  "doing"  such  as  stall 
recovery  and  crosswind  landings.  Problem  solving  with  procedural  knowledge  is 
more  of  a  stimuli-response  (or  learned  association  of  facts,  actions  and  results)  with 
which  certain  conditions  cause  certain  "learned"  (through  experience)  results.  All 
information  processing,  and  therefore  problem  solving,  involves  procedural 
knowledge,  i.e.,  tying  conceptual  knowledge,  environmental  inputs  and  situational 
memories  together  by  association  to  confront  new  situations. 

Significant  differences  exist  between  the  mental 
processing  performed  by  novices  and  experts  in 
how  they  approach  problem  solving.  In  order  to 
understand  these  differences  it  is  useful  to  define 
three  types  of  thinking  or  cognitive  processes: 
Algorithms,  Heuristics,  and  Production  Rules. 

Algorithms  are  procedures  guaranteed  to  result  in  the  correct  solution  of  a  problem. 
Algorithms  involve  a  systematic  and  thorough  search  of  the  conceptual  and 
procedural  knowledge  base,  an  examination  of  every  single  possibility  and 
evaluating  alternative  solutions  until  the  problem  is  solved.  For  example,  deciding 
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how  to  use  all  the  fuel  aboard  in  the  proper  sequence  to  insure  no  fuel  starvation  or 
out  of  balance  condition  during  a  max  duration  flight  involves:  knowledge  of  the 
aircraft's  capacities,  recent  fuel  consumption,  wind  velocity  at  various  altitudes,  etc. 
This  decision  requires  extensive  searching  of  stored  knowledge;  identifying, 
recalling  and  sorting  of  alternatives;  and,  selecting  the  correct  alternative.  This 
process  involves  forward  and  backward  searching,  setting  of  intermediate  goals, 
recognizing  similarities  and  differences,  drawing  analogies  and  evaluating  all 
alternatives.  This  type  of  thinking  or  problem  solving  is  very  accurate,  but 
extremely  time  consuming  and  often  not  practical  in  real  time  problem  solving. 

Heuristics  are  "rules  of  thumb"  which  offer  a  much  more  efficient  and  often  the 
operationally  useful  solution  of  a  problem,  but  not  necessarily  the  correct  solution 
in  every  case.  In  short,  heuristics  are  approximate  but  useful  problem  solving 
techniques.  In  aviation,  the  200  pound  male  passenger,  the  50  foot  FAA  tree,  the 
average  fuel  consumption  used  to  check  flight  progress,  and  flight  manual  values 
for  average  cruise  speed,  best  rate  of  climb,  etc.  are  all  heuristics. 

Production  rules  consist  of  conceptual  knowledge  combined  with  general  problem 
solving  procedures  to  create  new,  problem  specific  procedures.  Procedures  in  this 
sense  include:  flying,  navigating  and  ATC  as  well  as  heuristic  and  reasoning 
procedures. 

The  current  theory  of  problem  solving  is  that  a  novice  first  solves  problems  by 
weak,  domain  general,  heuristic  methods  (often  working  backwards  from  the  goal); 
successful  solutions  (when  repeated  frequently)  lead  to  the  development  of  domain 
specific  production  rules  and  the  beginnings  of  expertise;  as  these  rules  are  used 
more  and  more  often,  and  applied  to  many  situations,  they  result  in  automatic 
generation  of  specialized  production  rules  which  often  use  forward  inferencing  to 
progress  from  the  initial  problem  toward  a  solution  or  goal.  Relative  to  the  novice, 
the  expert  is  able  to  reach  the  correct  solution  more  quickly  and  efficiently. 

As  experience  is  gained  the  pilot  begins  to  develop  associative  problem  solving 
capabilities  for  enhanced  decision  making.  As  a  result  of  experience,  additional 
flight  training  and  possibly  a  knowledge  of  ADM  principles,  the  pilot  develops  a 
capacity  for  more  dynamic  thinking.  Information  is  stored  in  terms  of  schema 
based  upon  experience  and  training.  The  pilot  uses  pattern  recognition  and  dynamic 
interrelationships  among  objects,  situations  and  results  to  associate,  integrate  and 
interpret  related  knowledge  instead  of  the  static,  linear  thinking  of  the  novice.  This 
level  of  problem  solving  is  characterized  by  the  early  development  of  the  capabilities 
of  an  expert  decision  maker  in  that  certain  large  patterns  are  spontaneously 
recognized  (such  as  the  necessity  of  a  go-around  when  an  approach  is  too  high  and 
fast)  rather  than  requiring  a  conscious  search  of  conceptual  knowledge  and  a 
checklist  review  of  p*oblem  solving  steps. 

Since  experts  have  extensive  information  stored  as  schema,  this  information  can  be 
used  to  alter  procedures  in  real  time  (modify,  delete  or  expand).  Expert  pilots  can 
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create  new  production  rules  and  patterns  based  upon  unique,  previously 
unencountered  problem  characteristics.  This  capability  to  creatively  respond  to 
unique  problems,  novel  task  demands  and  chaotic  situations  identifies  the  highest 
level  of  expert  decision  making  (EDM).  At  this  level  the  pilot  has  become  an 
"adaptive"  expert.  Furthermore,  the  coordinated  use  of  cues  and  context  with  stored 
schema  is  believed  to  be  a  "major  mechanism"  used  by  experts  to  infer  unobserved 
or  unknown  elements  of  a  problem  in  knowledge  lean  or  chaotic  situations 
(Anderson,  1985). 

3.4  SOARing  Summary 

This  section  has  presented  a  unique  view  of  how  pilots  think  based  upon:  Expertise 
in  Action;  Information  Processing  System  characteristics;  and.  Problem  Solving 
processes.  The  Captain  of  the  UAL  flight  232  (used  as  an  example  of  successful 
decision  making  and  problem  solving  in  chaotic,  multiple  failures)  expressed  the 
opinion  that  PREPARATION  AND  EXECUTION  were  critical  factors  involved  in 
his  decision  making  and  problem  solving.  For  this  analysis  the  PREPARATION 
factor  was  limited  to  the  cognitive  aspects  of  Sensing  and  Organizing  information. 
The  EXECUTION  factor  was  similarly  limited  to  the  cognitive  aspects  of  Analyzing 
and  Responding  to  the  needs  of  the  situation. 

This  analysis  of  the  mental  aspects  of  preparation  and  execution  lead  to  the  coining 
of  the  term  Cognitive  SOARing  for  the  "Taming  of  the  Chaotic  Situation" 
demonstrated  in  several  of  the  recent  air  carrier  accident  "saves".  This  type  of 
problem  solving  behavior  has  been  recognized  and  identified  in  the  psychology  field 
as  Type  C  behavior  where  solutions  are  developed  under  time  pressured, 
ambiguous  circumstances.  Successful  training  of  Type  C  problem  solving  behavior 
has  been  demonstrated  in  other  fields  (Buffington,  1989,  Firestein  &  McCowan,  1988) 

as  discussed  in  Appendix  C.  The  transfer  of  this 
training  to  aviation  offers  potential  enhanced 
decision  making  training  for  pilots  and  should  be 
addressed  in  future  research. 


The  analysis  of  how  expert  pilots  think  was  then 
expanded  by  a  specific  review  of  the  Human 
Information  Processing  (HIP)  systems.  The  primary 
focus  of  this  discussion  was  on  the  capabilities  and 
limits  of  the  pilots  sensory,  short  and  long  term 
memory  systems  since  memory  is  the  fundamental  process  that  accounts  for 
differences  in  expert  and  novice  pilot  thinking  capabilities  .  Enhanced  memory 
training  to  develop  EDM  in  pilots  is  also  discussed  in  Appendix  B.  Finally,  the 
pilots  Problem  Solving  abilities  were  related  to  type  of  knowledge,  its  organization 
and  the  process  differences  between  novices  and  experts.  The  purpose  of  this 
discussion  was  to  expand  the  understanding  of  how  the  SOARing  processes  were 
accomplished  at  the  cognitive  level  and  to  specifically  identify  factors  which  could 
offer  opportunities  for  enhanced  training. 
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In  total,  12  information  processing  attributes  were  identified  as  offering  potential  for 
developing  future  Expert  Decision  Making  (EDM)  training  methods  and  materials. 
Other  fields  offer  existing  training  programs  for  enhancing  these  mental  attributes 
as  illustrated  in  Section  5.2  and  Appendix  C.  These  can  be  used  as  resources  to 
expedite  the  development  of  EDM  training  programs  for  pilots. 

3.5  DEVELOPING  EDM  TRAINING:  Initial  Aspects 

The  historical  approach  to  pilot  training  has  focused  on  three  basically  different 
categories  or  levels  of  piloting  tasks.  These  are:  skill  based  psychomotor  tasks,  rule 
based  perceptualmotor  tasks  and  knowledge  based  tasks.  The  skill  based  tasks  are 
those  which  have  been  learned  and  stored  as  psychomotor  routines.  These  tasks  can 
typically  be  performed  without  conscious  thought.  Actions  such  as  moving  the 
throttle  or  the  control  surfaces  are  examples  since  the  initial  conscious  intent  is 
translated  into  the  desired  actions  automatically. 

Rule  based  behaviors  are  those  "procedures"  which  have  been  learned  and  practiced 
throughout  early  training  in  response  to  both  normal  and  emergency  flying  tasks. 
These  procedures  consist  of  a  set  of  discrete  steps  necessary  to  successful  task 

completion.  Performing  a  takeoff  after  engine 
failure  at  V]  in  an  airliner,  engine  out 
procedures  on  a  light  twin  airplane,  and 
autorotation  in  a  helicopter  are  typical  of  these 
rule  based  tasks. 


Knowledge  based  tasks  are  those  for  which  no 
single,  specific  skill  or  procedure  h2S  been 
established.  Knowledge  based  behavior  is  the 
essence  of  decision  making  or  pilot  thinking 
and  will  provide  the  basis  for  understanding 
the  steps  necessary  to  develop  expert  decision  making  (EDM)  training  programs. 
The  way  in  which  expert  pilots  evaluate  information  from  the  environment,  use 
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memory  and  come  to  conclusions  will  be  described  in  order  to  differentiate  between 
what  is  currently  trained  and  what  is  perceived  as  needed  training  to  achieve  the 
next  breakthrough  in  the  reduction  of  human  error  accidents. 

These  tasks  require  the  information  processing  and  problem  solving  abilities 
analyzed  in  detail  in  the  previous  section.  The  basic  thrust  of  the  following  analysis 
will  be:  to  examine  differences  between  how  novice  and  expert  pilots  use  their 
knowledge  bases;  to  define  the  distinguishing  factors  of  expert  decision  makers;  to 
analyze  the  relationship  between  current  training  and  EDM  characteristics;  and 
finally,  to  describe  the  cognitive  processing  characteristics  common  to  adaptive 
experts,  Type  C  problem  solvers  and  chaos 
changing  pilots. 


4.0  EDM  AND  INTUITIVE  THINKING  AN  EXAMPLE 


The  analysis  of  EDM  has  shown  that,  basically, 
experts  structure  information  differently  in 
memory  and  integrate  new  information 
rapidly.  In  expert  pilots,  these  decision  making 
skills  have  been  demonstrated  to  provide 
keen,  quick  insight  and  almost  a  direct 
perception  of  the  proper  course  of  action.  This 
insight  results  in  decisional  behavior  based 
upon  intuition.  Aviation  author  Richard 
Collins  has  recognized  patience,  intuition, 
organization,  composure,  decisiveness  and 
coordination  as  critical  to  safe  flight  (Collins, 
1989).  He  describes  the  role  of  intuition  in 
aviation  as  follows:  "An  intuitive  pilot  is  less 
likely  to  be  surprised,  and  avoiding  surprises  is 
a  key  to  flying  well". 

4.1  EDM  Characteristics 

In  actuality,  the  experienced  pilot  or  expert 
decision  maker  sorts  the  available  information 
into  categories  or  "chunks"  more  meaningful 
to  the  goal  of  the  current  task,  for  example, 
continued  safe  flight.  Essentially,  experienced 
pilots  change  a  structured,  deductive  decision 
process  into  a  cued  recall  task  based  upon  the 
most  significant  chunks  of  information  and 
make  a  decision  based  upon  deductive 
reasoning  supplemented  by  an  inductive  or 
inferential  decision  making  style. 


The  United  Airlines  flight  81 1,  Boeing  747  accident,  in 
particular,  illustrates  how  experienced  pilots  rely  on 
intuition.  The  pilot  was  faced  with  handling  multiple 
emergencies  that  resulted  from  a  10  by  15  foot  hole  in 
the  fuselage  created  by  die  failure  of  a  cargo  door.  He 
was  faced  with  the  explosive  decompression  of  the 
cabin,  the  failure  of  two  engines  on  die  same  wing,  and 
a  malfunctioning  oxygen  system  as  he  reached  22,000 
feet.  The  normal  response  to  a  sudden  de  pressurization 
is  to  execute  a  power  dive.  But,  in  a  critical  decision, 
he  opted  to  be  cautious  and  descend  at  a  much  slower 
speed.  He  slowed  the  B747  to  as  close  to  stall  speed  as 
possible  to  keep  the  air  rushing  over  the  plane  from 
further  widening  the  hole  or  doing  more  damage  to  the 
wing.  The  trick  was  not  to  go  too  slow.  Since  the 
hole  changed  the  aerodynamics  of  the  plane,  he  really  did 
not  know  the  stall  speed  under  these  circumstances. 

Upon  reaching  4000  feet,  he  was  faced  with  insufficient 
time  to  dump  the  remaining  300,000  pounds  of  fuel  and 
flaps  that  could  only  be  extended  10  degrees  rather  than 
the  normal  20  degrees  for  a  two  engine  landing.  He 
decided  to  land  as  quickly  as  possible.  This  meant 
landing  at  195  miles  per  hour  vs.  the  normal  170  and 
landing  about  36,000  pounds  over  Boeing’s 
recommended  maximum  stress  load  of  564,000  pounds. 

With  fire  trucks  standing  and  ambulances  standing  by, 
the  Captain  made  what  some  flight  attendants  later  told 
investigators  was  one  of  the  smoothest  landings  they 
had  experienced.  In  this  case,  the  pilot’s  skill  and 
experience  combined  with  keen  intuition  and 
improvisation  to  keep  the  plane  from  crashing.  Tins 
example  further  illustrates  the  fast  access  to  an  expert 
knowledge  base  of  aerodynamic  principles,  his 
adaptability  and  the  autonomous  use  of  basic  flying 
skills  in  an  unpredictable,  abnormal  situation. 
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Therefore,  development  of  EDM  is  influenced  by  task  demands  encountered  in  the 
course  of  experience  and  training.  Experts  develop  the  capability  for  opportunistic 
planning  which  enables  them  to  revise  problem  representations  and  to  access 
multiple  possible  interpretations  of  a  situation.  These  multiple  patterns  are  quickly 
assessed  and  used  to  develop  an  "internal"  visualization  and  then  create  a  goal 
oriented  scenario  that  can  be  played  --  put  in  fine  detail  and  in  "slow-time"  --  to  a 
successful  solution.  In  contrast,  novices  are  less  flexible  and  slower. 

Experts  build  a  mental  representation  of  meaningful  relationships  in  a  situation. 
These  relationships  are  more  than  the  cognitive  knowledge  perceived  by  novices  in 
the  same  situation.  Long  familiarization  in  a  specific  field  of  knowledge  transforms 
the  experts  mental  representations  into  large  meaningful  patterns  which  when 
applied  to  working  situations  provide  the  fast  access  characteristic  of  intuitive 
responding  to  a  situation  (Bastic,  1982). 

Experts  also  possess  self-monitoring  or  metacognitive  abilities  that  are  not  present 
in  less  experienced  decision  makers  (Glaser,  1987).  Experts  develop  skilled  self- 
regulatory  processes  that  free  working  memory  for  higher  level  conscious 
processing.  These  capabilities  include:  planning  ahead,  efficiently  monitoring  one's 
time  and  attentional  resources,  and  monitoring  and  editing  one's  efforts  to  solve  a 
problem. 

These  observations  lead  to  an  interesting  contradiction.  Prior  to  the  development  of 
ADM  training,  it  was  widely  held  that  good  decision  making  could  only  be  learned 
through  experience.  The  15  years  of  ADM  work  has  contradicted  this  logic  by 
showing  that  some  aspects  of  good  decisions  can  be  trained.  However,  we  have  now 
realized  that  there  is  another,  more  subtle  impact  of  experience  on  decision  making 
in  which  the  steps  of  the  process  are  hidden  in  the  perceptual  and  procedural 
portion  of  human  information  processing.  Aviation  decision  making,  especially  in 
a  situation  (i.e.,  emergency)  characterized  by  rapid,  unexpected  changes  and  crises 
requires  an  adept  use  of  this  inferential  process  which  has  been  called  intuition. 

In  summary,  there  are  seven  basic  characteristics  of  Expert  Decision  Making  which 
are  extremely  relevant  to  the  desire  to  develop  EDM  training  materials  for  pilots. 
These  distinguishing  characteristics  have  been  identified  in  many  fields  of  expertise, 
but  they  have  particular  importance  in  the  current  analysis  of  how  pilots  use  their 
knowledge,  training  and  experience  to  overcome  the  adversities  associated  with 
everyday  flight,  i.e.,  how  pilots  think.  They  are  listed  in  the  figure  on  the  left  and 
summarized  for  ease  of  reference  as  follows: 
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A.  Superior  Short  Term  and  Long 
Term  Memory 


B.  Goal  Oriented 


C.  Fast  Access 


D.  Opportunistic  Planning 


E.  Adaptive 


F.  Self-Monitoring 


G.  Perceptual  Superiority 


The  material  presented  indicates  that  experts’  recall  seems  to  exceed  the 
limits  of  Short  Term  Memory.  Not  larger  than  other  humans’,  but 
autonomous  information  processing  of  many  of  their  skills  frees-up  greater 
storage.  Experts  seem  to  excel  in  Long  Term  recall  as  well.  Elaborations, 
associations  and  inferences  expedite  recall.  Cues  and  context  give  recall  the 
advantage  over  recognition. 

The  knowledge  of  experts  is  higltly  procedural  and  goal  oriented.  Concepts 
are  bound  to  procedures  for  their  applications  and  to  conditions  (contexts) 
under  which  these  procedures  are  useful.  This  functional  knowledge  is 
strongly  related  to  task  demands  and  goals  invaluable  in  problem  solving. 
Experts  can  solve  problems  quickly  and  accurately  for  three  reasons.  They 
are  faster  at  the  skill  based  tasks.  The  faster  skill  performance  frees-up 
working  memory  for  processing  other  aspects  of  the  problem.  They  can 
arrive  at  a  solution  without  conducting  extensive  search  of  memory. 
Experts  develop  the  capability  to  revise  production  rules  and  to 
simultaneously  access  multiple  possible  interpretations  of  a  situation.  The 
development  of  this  capability  is  influenced  by  task  demands  and 
experience. 

There  are  both  routine  and  adaptive  experts.  Either  type  is  outstanding  in 
terms  of  speed,  accuracy  and  automaticity  of  their  decision  making.  Either 
type  can  construct  mental  models  convenient  for  their  tasks.  While  both 
are  very  confident  in  the  execution  of  their  solutions,  routine  experts  have 
somewhat  limited  capabilities  in  dealing  with  new  or  ill  structured 
problems.  Adaptive  experts  possess  the  ability  to  creatively  respond  and 
develop  solutions  to  ill  structured  or  ambiguous  new  problems  with  some 
reasonable  chance  for  a  successful  outcome. 

The  greater  knowledge  bases  and  different  knowledge  representations  of 
experts  allow  more  time  to  predict  problem  difficulty  on  the  basis  of 
underlying  principles  and  more  time  to  monitor  accurately  how  they  should 
allocate  their  time  for  solving  problems.  This  solution  monitoring  or  self- 
regulatory  capability  are  manifested  by  experts  developing  automaticity  in 
performing  basic  operations,  by  their  allocation  of  attention,  and  by  their 
sensitivity  to  informational  feedback. 

Experts  have  the  ability  to  perceive  large  meaningful  patterns  due  to  the 
organization  of  their  knowledge  base.  This  organization  is  based  on  long 
familiarization  (experience  and  training)  in  a  field.  The  pattern  recognition 
and  recall  occurs  so  rapidly  that  it  takes  on  the  characteristics  of 
instantaneous  insight  or  intuition. 


expert  koakdikjq 


SUPERIOR  MEMORY 


PERCEPTUAL 

SUPERIORITY 


SH  *•  MONITORING 


GOAL  ORIENTED 


FAST  ACCESS 


ADAPTIVE 


OPPORTUNISTIC 
PLAN  NINO 


This  project  was  designed  to  search  for 
enhanced  cognitive  training  solutions  in 
order  to  resolve  the  contradiction  of  current 
ADM  training  .  The  research  described  in 
previous  sections  has  provided  the 
foundation  for  those  enhancements  in 
memory  training,  problem  solving  training 
and  expertise  training.  The  characteristics 
documented  for  expertise  has  led  to  the 
exploration  of  the  role  of  intuition  in  the 
way  pilots  respond  to  normal,  emergency 
and  sometimes  novel  (or  chaotic)  situations. 


The  following  analysis  is  intended  to  provide  additional  refinements  to  the 
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development  of  second  generation  ADM  training  materials  and  if  possible  lead  to 
enhanced  or  Expert  Decision  Making  training  which  can  be  learned  without  sole 
reliance  on  experience  in  an  aircraft. 


4.2  Intuition  Characteristics 

Intuition  has  been  called  "one  of  the  most  important  faculties  of  man"  (Lorenz, 
1951).  All  of  us  have  intuitive  abilities.  In  various  degrees  they  pervade  everything 
we  do.  They  range  from  the  scientist's  Eureka  experiences  to  the  day-to-day  hunches 
and  "feelings"  which  guide  more  common  decisions.  The  literature  (Fishbein,  1972) 
discusses  two  types  of  intuition: 

a.  Primary  -  basic  instincts  constructed  without  any  systematic  process 

(we  are  not  addressing  this  type  of  intuition) 

b.  Secondary  -  academically  learned  (or  knowledge  based)  intuitions  (e.  g., 

lift/drag  and  power/ pitch  airmanship  decisions 

Operationally,  the  secondary  type  of  intuition  is  critically  important  to  EDM. 

The  most  meaningful  definition  of  intuition  (Bastic,  1982)  for  the  purposes  of  this 
discussion  is: 

"Knowledge  based  on  experiences  and  acquired  through  sensory  contact". 

In  this  context,  intuition  can  be  viewed  as  an  inferential  process  similar  to  ordinary 
perceptions.  That  is,  intuition  does  not  advance  in  a  linear  set  of  steps.  It  tends  to 
involve  responding  to  situations  based  seemingly  on  an  implicit  perception  of  the 
whole  problem.  As  with  other  procedural  knowledge,  the  pilot  arrives  at  an  answer 
and  acts  on  it  but  can  have  little  or  no  awareness  of  the  process  by  which  he  reached 
it.  Furthermore,  this  knowledge  cannot  be  verbalized  beforehand  as  it  is  activated 
by  the  cues  and  context  of  the  situation.  Nevertheless,  intuition  is  a  rational  skill 
and  one  that  is  sometimes  necessary  for  pilots  to  use.  Intuition  is  an  extension  of 
the  logical  problem  solving  process  and  one  in  which  the  steps  of  the  process  are 
hidden  in  the  subconscious  portion  of  the  brain. 

The  basic  characteristics  of  intuition  are  defined  in  Appendix  B.  These 
characteristics  are  listed  on  the  following  page  with  those  properties  most  applicable 
to  how  expert  pilots  think  indicated  in  bold  type.  These  characteristics  are  provided 
as  more  definitive  descriptors  of  intuition.  They  are  not  independent  or  necessarily 
observable.  The  apparent  relationship  between  these  characteristics  of  intuition  and 
Expert  Decision  Making  listed  offers  additional  opportunities  for  EDM  training 
material  development. 
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The  detailed  analysis  (Appendix  A)  of  Expert  Decision  Making  points  out  that 
intuition  is  an  integral  part  of  the  "autonomous"  stage  of  how  pilots  think. 
Intuition  requires  extensive  knowledge  of  the  subject  matter  and  basic  principles  of 
aeronautics  and  extensive  experience  in  aviation.  It  is  the  fast  access  to  large 


EttVTXPFLlCATlOS 


lSttlfldN'Pk'OP'mE? 

1 .  QUICK,  IMMEDIATE,  APPEARANCE 

2.  EMOTIONAL  INVOLVEMENT 

3.  PRECONSCIOUS  PROCESS 

4 .  NOT  ANALYTICAL  PROBLEM  SOLVING, 
LOGIC  OR  ABSTRACT  REASONING 

5.  INFLUENCED  BY  EXPERIENCE 

6 .  EMOTIVE  NOT  TACTILE 

7.  ASSOCIATED  WITH  CREATIVITY 

8 .  ASSOCIATED  WITH  EGOCENTRJCTTY 

9 .  INTUITION  NEED  NOT  BE  CORRECT 

10.  SUBJECTIVE  CERTAINTY 
OF  CORRECTNESS 

1 1 .  RBCENTERING 

1 2 .  EMPATHY.  KINESTHETIC  OR  OTHER 

1 3 .  INNATE,  KNOWLEDGE  OR  ABILITY 

1 4 .  PREVERBAL-CONCEPT 

15.  GLOBAL  KNOWLEDGE 

16.  INCOMPLETE  KNOWLEDGE 

1 7 .  HYPNOGOGIC  REVERIE 

18.  SENSE  OF  RELATIONS 

19.  DEPENDENCE  ON  ENVIRONMENT 

20 .  TRANSFER  AND  TRANSPOSITION 


AUTONOMOUS  PROCESSING 


KNOWLEDGE  ORGANIZATION  & 
ELABORATION 

ADAPTIVE  EXPERT 


PILOTS  ARE  SELF-ASSURED  & 
OPTIMISTIC  DECISION  MAKERS 


LARGE  MEANINGFUL  PATTERNS 
ILL-STRUCTURED.  AMBIGUOUS  AND 
KNOWLEDGE  LEAN  PROBLEM  SOLVING 

PATTERN  RECOGNITION 

CUES  AND  CONTEXT  OF  SITUATIONS  OR 

DECISION  TASK  DEMANDS 


meaningful  patterns  of  knowledge  that  "appears"  intuitive  to  the  observer.  In 
actuality,  the  autonomous  decision  maker's  ability  to  respond  creatively  to  novel 
situations  or  ill-structured  problems  relies  on  his  knowledge  structure  and 
experience.  In  fact,  intuitive  mental  processes  seem  to  be  common  whenever  the 
situations  become  too  complex  for  logical  analysis.  In  this  sense,  intuition  is  a 
superior  form  of  cognition. 


5.0  EDM  TRAINING  DEVELOPMENT 

Pilot  decision  making  experience  is  currently  developed  over  many  years  and 
thousands  of  hours  of  learning  and  exposure.  As  with  other  fields  of  expertise,  the 
pilots  decision  making  abilities  continue  to  develop  throughout  his  career  and 
training  as  a  function  of  practice.  That  is,  his  conceptual  knowledge,  procedural 
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knowledge  and  flying  skills  continue  to  expand  and  to  provide  more  associations, 
elaborations,  inferences  and  patterns  with  which  problems  can  be  solved.  As 
pointed  out  in  the  literature  (Chi,  Glaser  &  Farr,  1988  and  Ericsson,  1991),  studies 
have  been  carried  out  in  many  other  fields  which  illustrate  the  need  for  many  years 
of  full  time  experience  and  training  to  achieve  the  level  of  Expert.  Third  generation 
theories  of  expertise  suggest  that  more  than  10  years  experience  may  be  required  to 
reach  the  level  of  Adaptive  Expert  (Holy oak,  1990). 

Furthermore,  it  is  a  common  observation  in  aviation  as  well  as  research  on  Expert 
Decision  Making  that  in  the  course  of  acquiring  expertise,  plateaus  in  development 
are  observed  which  apparently  indicate  shifts  in  understanding  and  the  change  from 
knowledge  based  processing  to  autonomous,  procedurally  dominant  processing. 

This  has  led  to  the  conclusion  that  although  both  novices  and  experts  have  similar 
capacities  for  processing,  the  outstanding  performance  of  experts  derives  from  their 
knowledge  based  on  "experiences"  (that  is  familiarity  with  contexts  and  cues)  which 
facilitate  situational  awareness,  the  anticipation  of  risk  and  the  task  oriented 
decision  making  required  to  reduce  or  manage  the  risk. 

Keeping  in  mind  the  importance  of  skill  based,  rule  based  and  knowledge  based 
training  on  the  content  and  organization  of  a  pilot's  knowledge,  this  section 
correlates  the  more  specific  aspects  of  current  pilot  training  with  the  characteristics 
identified  for  Expert  Decision  Making.  The  following  table  correlates  experience 
based  pilot  decision  making  and  problem  solving  characteristics  with  the  (EDM) 
capabilities  identified  in  Section  4.1. 


CURRENT  TRAINING  EXPERT  DECISION  MAKING  CHARACTERISTICS 

ELEMENTS  (NOTE:  'I  =  PRIMARY  RELATIONSHIP  &  0  =  SECONDARY) 


SUPERIOR 

GOAL 

FAST 

OPPORTU¬ 

ADAPTIVE 

SELF-  MON¬ 

PERCEPTUAL 

MEMORY  ORIENTED  ACCESS 

NISTIC 

ITORING 

SUPERIORITY 

RELIES  ON  PRACTICE/  REPETITION 

V 

0 

V 

CONTEXT  (TASK  DEMAND)  BASED 

V 

V 

0 

0 

SKILL  BASED 

V 

V 

RULE  (PROCEDURES)  BASED 

V 

0 

AUTOMATIC 

V 

V 

V 

KNOWLEDGE  BASED 

0 

V 

V 

GOAL  ORIENTED 

V 

V 

POSITIVE  APPROACH  &  OPTIMISTIC 

0 

0 

V 

SITUATION  AWARENESS 

0 

V 

V 

V 

AERONAUTICAL  DECISION  MAKING 

0 

V 

0 

V 

RISK  MANAGEMENT 

0 

V 

V 

JUDGMENT 

0 

V 

V 

0 

V 

to 


The  purpose  of  this  table  is  twofold.  First,  it  illustrates  the  important  correlation 
between  the  factors  involved  in  EDM  and  those  which  are  currently  trained. 
Second,  it  indicates  that  the  Expert  Decision  Makers  more  complex  characteristics: 
opportunistic  planning,  adaptivity,  self-monitoring  and  perceptual  superiority  rely 
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on  the  cognitive  characteristics  or  thinking  capabilities  of  autonomous  information 
processing.  These  were  pointed  out  in  Section  3.0  as  creative  problem  solving  (i.e., 

opportunistic  planning  and  adaptivity), 
judgment  and  intuition  or  insight.  The 
discussion  which  follows  expands  on  these 
characteristics  in  order  to  complete  the 
delineation  of  those  areas  which  could 
comprise  enhanced  or  expert  pilot  decision 
making  training  programs. 


5.1  EDM  Training  Materials,  Methods  and 
Tools) 

Now  that  we  have  reviewed  the 
characteristics  of  how  experts  think  and  how  these  same  processes  apply  to  expert 
pilot  decision  making,  we  have  discovered  that  the  important  problem  solving 
activities  become  more-and-more  automatic.  In  fact,  the  speed  and  accuracy  become 
so  dominant  that  the  experts  "intuitively  respond"  even  as  tension  and  problem 
complexities  mount  and  go  beyond  the  "practiced"  emergencies.  More  importantly, 
by  listing  the  gradual  build-up  of  properties  important  to  EDM: 

-  attention,  short  term  memory,  long  term  memory 

--  large  meaningful  patterns  of  knowledge,  adaptivity,  self-monitoring 

--  influenced  by  experience,  self-assured,  dependent  on  environment 

we  have  illustrated  that  expertise  and  intuition  are  not  separate  phenomena,  but  as 
noted  by  Nobel  Laureate  Herbert  Simon  "aspects  of  the  same  thing"  (Benderly,  1989). 
Dr.  Simon,  professor  of  computer  science  at  Carnegie  Mellon  University,  is  one  of 
several  researchers  including  Dr.  Robert  Glaser,  psychology  professor  at  the 
University  of  Pittsburgh,  Dr.  Robin  Hogarth,  Center  for  Decision  Research  at  the 
University  of  Chicago,  and  their  associates,  who  reject  the  idea  that  intuition  is  an 
inexplicable  gift.  These  noted  scientists  explain  that  intuition  is  a  predictable 
product  of  the  way  experts  think.  Even  more  importantly,  their  work  suggests  that 
intuition  may  be  an  ability  that  individuals  can  work  toward  and  organizations  (i.e. 
airlines,  training  schools,  NTSB,  FAA,  etc.)  can  foster. 

The  principles  of  this  research  have  been  thoroughly  described  in  the  first  four 
sections  of  this  report.  The  next  step,  trying  how  to  figure  out  methods  to  train  EDM 
and  maybe  a  little  intuitive  ability,  will  be  addressed  in  the  following  pages. 
However,  the  physical  limits  of  time  and  funding  will  not  permit  great  detail  or 
rigor  in  the  methods  outlined.  The  intent  of  this  discussion  is  twofold:  to  introduce 


WHAT  DO  WE  REMEMBER  ? 

•  10%  OF  WHAT  WE  READ 

•  20%  OF  WHAT  WE  HEAR 

•  30%  OF  WHAT  WE  SEE 

•  50%  OF  WHAT  WE  HEAR  &  SEE 

•  70%  OF  WHAT  WE  SAY 

•  90%  OF  WHAT  WE  SAY  &  DO 


REFERENCE:  VERVALDI,  C  EL,  1S9L  MANAGEMENT  AND  THE 
TECHNICAL  PROFESSIONAL.  GULF  PUBLISHING 
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the  methods  which  are  more  fully  described  in  Appendices  C,  D  and  E;  and,  to 
provide  direction  and  focus  for  future  research  which  appears  to  offer  the  next  great 
improvement  in  reducing  decision  making  errors. 

In  striving  to  achieve  this  goal,  it  is  necessary  to 
depart  from  conventional  aviation  training  in 
several  respects.  First,  new  paradigms  for 
developing  decision  making  training  need  to  be 
considered  without  the  limitations  of  current 
training  and/or  testing  syllabi.  The  EDM 
paradigms  need  to  break  with  the  conventional, 
linear,  checklist  ADM  training  in  order  to  make  the  necessary  revolutionary  strides 
in  safety  improvements.  These  should  include  the  recent  developments  in 
modeling  chaotic  processes,  i.e.,  processes  that  are  non-linear  and  very  sensitive  to 
initial  conditions.  Second,  several  non-aviation  fields  need  to  be  explored  where 
applicable  types  of  training  either  already  exist  or  have  been  evaluated  from  a 
research  perspective  (see  Appendix  C  examples).  Third,  decisions  about  the  training 
media  need  to  be  considered  (Appendix  D. 

The  reliance  of  both  expertise  and  intuition  on  "experience",  the  environmental 
stimuli,  and  the  cues  and  context  of  the  situation  make  this  decision  extremely 
relevant.  The  fields  of  Activity  Based  Learning  and  Interactive  Computer  or  Video 
Based  Training  (CBT/VBT)  devices  offer  near  term  cost  effective  alternatives  which 
are  attractive  to  both  airline  and  general  aviation  pilots  alike.  In  1991,  one  of  the 
major  U.  S.  air  carriers  invested  $4.5  million  in  interactive  technology  to  train  about 
1000  pilots  per  day  as  well  as  software  for  training  flight  attendants,  cargo-handlers 
and  ticket  agents  (Todd,  1991).  Yet,  none  of  this  training  addressed  the  important 
areas  of  decision  making  training,  the  important  elements  of  cognitive  SOARing  or 
problem  solving  expertise.  Furthermore,  this  relatively  mature,  and  inexpensive 
type  of  training  is  not  available  to  the  general  aviation  community  except  on  a 
limited  basis  (See  CEPET  Section  5.3. A  and  Appendix  D.l).  The  kinds  of  applications 
which  are  and  are  not  suitable  for  CBT  will  be  discussed  when  the  available  systems 
are  reviewed. 


ENHANCED  TRAINING 
CONCEPTS 


•  NCW  PAftAQOUS  POfl  EXPERT  DECISION  UAKMO 

•  CHAOTIC  PROCESS  UOOCLJNO  A  HO  NOVEL  CUES 

•  APPLICATIONS  OP  ACTIVITY  BASED  LEAAMNQ 

•  INTERACTIVE  METHODS  AMO  TOOLS 


5.2  Related  Cognitive  Training  Methods 


COGNITIVE  TRA3N3NG 
EXAMPLES 


1.  KAftllUN-MCfTIII,  PAWS  •  IRAiN  OYM 

1  TYWI  0  -  CHAOS  CHAMOKJI1  OR  UtPtAT  MtOILEM 
90LVKM 


Three  examples  of  existing  non-aviation 
cognitive  training  program  opportunities 
(programs  which  may  offer  some  aviation 
applications  and  potential)  are  reviewed  in 
Appendix  C.  These  include: 


1  CaNtU  KM  OKtOOM  KUUMH  U.  Of  CMCAOO 

-THOMOHTTiMAia-  •  BRAIN  GYM  —  A  program  to  improve 

memory,  logic,  attention  and  organizational  ability.  The  month  long  course  has 
reported  the  following  results: 

1.  a  32.5%  improvement  in  mental  agility 

2.  a  47.3%  improvement  in  visual/spatial  memory 

3.  a  20.1%  improvement  in  verbal  memory 


•  TYPE  C  PROBLEM  SOLVING  --  This  program  has  trained  thousands  of 
individuals  in  effective  problem  solving,  i.e.,  identifying  the  right  problem  and 
implementing  the  right  solution  at  the  appropriate  time.  It  involves  a  two  phase, 
six  step  method  employing  a  stepwise  problem  solving  model  that  uses  forward 
inferencing  in  the  information  processing  solution. 

•  THOUGHT  TRIALS  --  A  program  to  improve  the  decision  related  skills  of 
association,  retention  and  creative  problem  solving.  The  thought  trial  program  is 
based  upon  the  theory  that  creative  thinking  and  problem  solving  involves  the 
same  principles  as  trial-and-error  learning.  By  providing  individuals  with  a  very 
large  number  of  associative  problem  solving  paths,  the  thought  trial  process  builds 
the  individuals  ability  to  provide  foresight  in  new  situations.  Furthermore,  the 
paths  which  lead  to  correct  solutions  are  retained  as  strong  connections  since  the 
overt  behavior  of  "finding"  a  solution  is  accompanied  by  the  feelings  of 
accomplishment  and  discovery. 


5.3  Existing  EDM  Tools 

This  section  provides  three  examples  of  Expert  Decision  Making  tools  which  can  be 
used  for  either  training  decision  making  skills,  evaluating  cognitive  performance  or 
in  one  case  as  a  screening  tool  to  determine  a  priori  if  a  student  pilot  has  good 
decision  making  skills.  These  tools  were  selected  due  to  their  tailored  development 
in  the  aviation  human  error  reduction  field.  However,  they  are  not  intended  to 
represent  the  recommended  choices  or  even  a  prioritized  list.  Rather,  they  axe  to 
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EDM  TRAINING  TOOLS 


represent  the  type  of  interactive  tools  which  can 
expedite  and  enhance  how  pilot's  think  and  solve 
problems. 


of-the-art  technology  that  could 
the  pilot  population. 

A.  CEPET 

The  Cockpit  Emergency  Procedures  Expert  Trainer  (CEPET)  is  a  personal  computer- 
based  training  program  which  contains  expert  pilot  knowledge  about  cockpit 
emergency  procedures  and  pilot  decision  making.  CEPET  was  developed  by  Bell 
Helicopter  Textron,  and  is  currently  available  for  four  helicopter  models:  the  Bell 
206B3, 206L3,  212,  and  412.  The  CEPET  knowledge  based  software  addresses  good  and 
bad  decision  making,  attention,  dual  task  loading,  fatigue,  emotion, 
overcompensation  and  risk  taking. 

The  program  has  several  modules  dealing  with  flight  planning,  flight  scenarios, 
situational  awareness  and  stress  management.  Questions  and  answers  are  addressed 
interactively  in  the  areas  of  helicopter  systems,  normal  and  emergency  procedures, 
weather.  Air  Traffic  Control  and  airspace.  Some  of  the  modules  start  a  flight 
scenario  with  an  initial  set  of  cues.  The  student  pilot  or  recurrent  trainee  identifies 
the  problem  and  chooses  an  appropriate  action.  Other  modules  ask  the  pilots 
questions  about  some  of  the  more  serious  emergencies  that  would  call  for 
premature  landings.  As  the  pilot  responds  to  the  various  scenarios  and  questions 
with  his  aeronautical,  systems  or  helicopter  knowledge,  the  program  helps  in 
identifying  the  probable  causes  of  the  various  emergencies  and  recommends 
appropriate  actions.  The  pilot  can  practice  his  ability  to  recognize  problems  with 
only  partial  cues  and  choose  the  correct  solutions.  A  more  detailed  description  is 
provided  in  Appendix  D.l 

B.  WOMBAT 

The  Wondrous  Original  Method  for  Basic  Airmanship  Testing  (WOMBAT)  is  a 
pilot  decision  making  evaluation  and  selection  tool  designed  to  screen  pilots  for  the 
basic  abilities  or  attributes  credited  to  operationally  effective  pilots.  In  this  sense,  it 


Additional  tools  may  be  available  of  which  the 
author  is  not  aware.  Also,  the  type  of  interactive 
and  evaluative  efforts  discussed  here  are  meant  to 
be  representative  of  the  kind  of  cost  effective,  state- 
bring  EDM  training  to  a  much  broader  segment  of 


•  CIPIT  .  interactive  computer  or  video 

TRAININO  OtVICi 


•  WOMBAT  -  WONOftOUS  ORIGINAL  MfTHOO  FOR 
BA  tic  AIRMANSHIP  TRAINING 


•  CRM  ON  LOS  .  COONITIY!  EFFECTIVENESS 

MARKERS 
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offers  a  potential  development  path  for  more  sophisticated  EDM  training  devices. 
WOMBAT  was  initially  developed  for  the  U.  S.  Air  Force  by  Dr.  Stanley  Roscoe  and 
his  research  team  at  ILLIANA  Aviation  Sciences  based  in  Las  Cruces,  New  Mexico. 

WOMBAT  is  a  microprocessor  based  system  w’hich  integrates  generic  information 
processing  algorithms  with  new  pictorial  codes  and  new  decisional  paradigms.  In 
this  way,  scenarios  are  generated  which  require  dynamic  information  sensing  and 
processing  with  changing  outputs  and  new  opportunities  for  more  decisions  and 
problem  solving.  Pilot's  earn  or  lose  points  at  varying  rates  by  the  way  in  which 
they  analyze  the  situation,  infer  the  rules,  invest  attentional  resources  and  capitalize 
on  opportunities  to  score  when  serendipitous  occasions  are  presented.  A  more 
detailed  description  is  provided  in  Appendix  D.2. 

C.  Cognitive  Effectiveness  Markers 

A  very  different  type  of  evaluation  tool  is  currently  being  developed  for  multi-crew 
decision  making  and  problem  solving.  In  the  air  line  environment,  the  1990s  have 
become  a  period  of  focusing  on  crew  resource  management  (CRM).  That  is,  a  time 
when  the  true  team  resources  are  described  not  only  in  terms  of  the  cockpit  crew  but 
include:  the  cabin  crew,  dispatch  and  maintenance  personnel.  This  team 
performance  has  been  recognized  as  a  critical  part  of  insuring  safe  and  efficient 
flight. 

Consequently,  Dr.  Robert  Helmreich  of  the  FAA/NASA/University  of  Texas  Crew 
Performance  Project  has  been  developing  a  set  of  observable,  evaluatable 
performance  markers  to  accomplish  this  task  (Butler,  1992).  He  calls  them 
Behavioral  Performance  Markers.  The  emphasis  in  this  assessment  process 
includes  the  integration  of  cognitive,  interpersonal  and  technical  skills  evaluation 
of  the  flightcrew's  performance.  The  airlines,  the  FAA's  ARD-200  and  AFS-210  have 
all  beer,  involved  in  this  developmental  process  (Lofaro,  1992).  In  fact,  ATA 
Working  Group  quarterly  meetings  have  been  held  for  the  past  two  years  with  the 
results  that  the  Cognitive  and  Interpersonal  markers  have  been  included  in  the 
latest  draft  FAA  Advisory  Circular  on  CRM  (Anon,  AC  120-51  a,  1992).  Although 
this  work  is  still  maturing,  it  has  had  wide  spread  distribution.  Several  major  air 
carriers  feel  that  it  is  readily  adaptable  to  extensive  crew  and  single  pilot  cognitive 
effectiveness  evaluations  Assessment  of  its  usefulness  is  expected  to  take  place 
primarily  in  the  air  line  industry  as  outlined  in  the  FAA's  requirements  for 
Advanced  Qualification  Programs. 

5.4  Promising  EDM  Training  Programs 

Based  on  the  research  from  Phase  f  and  II  of  this  project  on  Expert  Decision  Making, 
three  promising  EDM  training/testing  methods  are  suggested  as  applicable  to  future 
pilot  training  in  decision  making.  These  are: 
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•  Metacognition  Training  (under  evaluation  using  novice  pilots  at  Massey 

University  in  New  Zealand) 

•  Intuitive  Decision  Making  Testing  (evaluated  intuitive  decision  making  skills  of 

over  3000  managers) 

•  Cognitive  SOARingTesting  and  Training  (developed  as  a  part  of  this  FAA  project) 

The  first  two  methods  have  been  empirically  tested  and  offer  encouraging  examples 
of  what  can  be  done.  The  third  method  is  a  proposal  for  future  EDM  testing  and 
training  that  could  carry  this  FAA  research  into  expert  decision  making  forward  to 
the  verification  and  validation  stage. 

A.  Metacognition  Training 

The  School  of  Aviation  at  Massey  University  in  New  Zealand  has  been  developing 
and  evaluating  training  of  ab  initio  pilots  in  the  cognitive  skills  of  experts  since  1990 
(Hunt,  1991).  Their  pilot  training  program  assesses  each  incoming  student  pilot  for 
their  information  processing  styles.  The  learning  profile  derived  from  this 
assessment  becomes  the  basis  for  working  with  students  to  develop  their  decision 
making,  problem  solving  and  metacognitive  (self-monitoring)  abilities.  Specific 
emphasis  is  placed  on  self-monitoring  and  self-evaluation  of  the  decision  making 
process.  This  program  is  summarized  here  and  reviewed  in  detail  in  Appendix  E  to 
illustrate  what  can  be  accomplished  and  what  may  be  an  interesting  avenue  for 
rther  research  and  development  by  the  Federal  Aviation  Administration. 

e  training  program  at  Massey  University  recognizes  the  importance  of  strategic 
formation  processing  in  pilot  performance  of  both  normal  and  emergency  tasks, 
addition  to  the  usual  focus  of  ab  initio  training  on  manipulative  skills  and 
aditional  content  such  as  navigation,  aircraft  systems  management,  etc.  their 
rogram  provides  training  in  the  broader  context  of  information  processing.  This 
raining  is  based  upon  the  recognition  that  the  more  common  causes  of  pilot  error 
lave  an  underlying  information  processing  function  and  that  if  the  trainee  pilots 
:an  improve  the  efficiency  and  effectiveness  of  their  "thinking",  and  thus  improve 
their  monitoring  and  evaluative  functions,  then,  their  overall  performance  as  pilot- 
in-command  will  improve. 

Due  to  the  dilemma  of  developing  a  course  for  the  average  student  while  trying  to 
teach  expert  decision  making,  it  was  necessary  to  develop  a  compromise  between 
individual  attention  and  group  learning.  Therefore,  the  course  begins  with  an 
overview  of  information  processing  and  the  factors  which  can  influence  effective 
learning.  The  basic  course  syllabus  provides  the  students  with  the  appropriate 
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knowledge  and  procedures  to  interpret  and  understand  their  own  "thinking"  or 
decision  making  processes. 


These  include  the  Approach  to  Learning,  the  Self  Concept  and  Metacognition.  Very 
early  in  the  course  the  pilots  are  introduced  to  time-pressured  and  dynamic  decision 
situations.  They  are  required  to  identify  actions,  set  priorities  and  apportion  their 
time  to  specific  tasks.  They  are  encouraged  to  continually  review  and  evaluate  their 
decisions. 

The  following  rating  factors  are  used  in  the  evaluation  of  the  student  pilot 
performance  by  the  instructor.  First,  they  are  rated  on  their  ability  to  analyze 
objectives  and  evaluate  the  content  of  the  task  or  situation.  Second,  they  are 
evaluated  on  the  appropriate  application  of  conceptual  knowledge  (facts)  or 
procedural  knowledge  (rules)  used  to  address  the  situation.  Third,  they  are 
evaluated  on  the  remembering  and  retention  phase.  For  each  objective  identifying 
the  required  processing  level  helps  the  student  to  choose  the  appropriate  cognitive 
strategies  for  learning. 

Massey's  first  information  processing  course  has  been  completed  but  a  final 
assessment  of  the  successes  and  failures  has  not  been  reported  at  this  time. 

However,  preliminary  data  presented  at  the  1991  Sixth  International  Symposium  on 
Aviation  Psychology  showed  some  promising  indications.  These  are: 

"At  the  start  of  the  course  students  were  able  to  identify  between  0-5  strategies  for  remembering  type 
learning  tasks,  and  0-3  for  deep  learning  tasks.  Approximately  45%  of  the  strategies  they  suggested 
were  inappropriate  to  the  nature  of  the  task,  After  six  months  into  the  program,  information  from  at 
least  30  study  sessions  per  student  showed  that  they  are  able  to  write  good  instructional  objectives, 
determine  the  content  type  and  process  level  of  the  learning  task  and  select  strategies  appropriate  to 
the  task.  The  number  of  strategies  identified  ranged  from  5-9  for  remembering  tasks  and  14  to  19  for 
deep  or  meaningful  tasks.  These  figures  suggest  students  have  an  increased  awareness  and  range  of 
strategies,  more  appropriate  usage  and  are  able  to  use  a  greater  number  of  strategies  per  task." 

In  the  U.  S.,  ADM  work  (Section  1.0) 
has  shown  that  the  quality  of 
decision  making  can  be  enhanced 
with  the  appropriate  development 
of  cognitive  skills  in  a  classroom 
setting.  It  appears  that  the  high 
quality  of  information  processing 
skills  being  taught  at  Massey 
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University  may  offer  an  attractive  growth  direction  for  decision  making  training  in 
the  U.  S. 


B.  Intuitive  Decision  Making  Testing 

In  the  1980s,  the  application  of  expertise  and  intuitive  decision  making  gained 
acceptance  in  guiding  executive  decision  making  for  a  wide  range  of  industries.  A 
successful  field  testing  of  intuitive  decision  making  skills  of  over  3000  managers 
throughout  the  U.  S.  in  a  wide  variety  of  organizational  settings  was  a  part  of  that 
acceptance  effort  (Agor,  1986).  These  settings  included  business,  government, 
education,  military  and  health  and  a  wide  range  of  management  levels.  This 
experience  and  the  positive  results  once  again  (as  with  the  Brain  Gym,  Thought 
Trials,  etc.)  suggests  an  opportunity  exists  to  transfer  developments  in  appropriate 
testing  or  training  methods  from  other  fields  to  aviation  without  repeating  all  of  the 
learning  curve  work  at  great  financial  and  time  costs. 

When  the  executives  who  scored  in  the  top  10  percent  in  intuitive  ability  are  tested, 
the  results  overwhelmingly  indicate  that  these  executives  do  use  their  autonomous, 
adaptive  procedural  reasoning  to  guide  their  most  important  decisions.  Not  only  do 
the  vast  majority  of  these  top  executives  admit  that  they  use  intuition  to  help  guide 
their  most  important  decisions,  but  they  go  on  to  specify  the  situations  and  contexts 
in  which  they  find  their  intuition  is  most  helpful: 

•  When  there  is  a  high  level  of  uncertainty. 

•  When  there  is  little  previous  precedent. 

•  When  the  variables  are  not  very  scientifically  predictable. 

•  When  "facts"  are  limited  (i.e.,  knowledge  lean  situations). 

•  When  "facts"  do  not  clearly  point  the  way  to  go  (ambiguous). 

•  When  there  are  several  plausible  alternative  solutions  to  choose 

from,  with  good  arguments  for  each. 

•  When  analytical  data  are  of  little  use  (e.g.,  new  trends  are  emerging) 

•  When  time  is  limited  and  there  is  pressure  to  be  right. 

It  is  also  significant  to  note  that  when  these  top  managers  intuitively  "know"  they 
have  reached  the  correct  decision,  they  share  a  "consensus  set"  of  feelings  that  tell 
them  so:  a  seare  of  excitement-almost  euphoria;  a  total  sense  of  commitment;  a 
feeling  of  total  harmony;  warmth  and  confidence;  a  burst  of  enthusiasm  and  energy 
that  "this  is  the  right  solution."  Pilot's  surviving  abnormal  emergencies  and 
chaotic,  multiple  failures  have  reported  very  similar  feelings.  Alternatively,  when 
they  sense  an  impending  decision  may  be  an  incorrect  one  or  that  they  need  to  take 
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more  time  to  adequately  process  the  cues  and  information  they  are  receiving,  these 
managers  speak  of  feelings  of  anxiety,  mixed  signals,  discomfort,  or  an  upset 
stomach. 

Clearly  there  are  parallels  in  the  executives  use  and  need  for  intuition  to  what  we 
have  discussed  for  expert  pilot  decision  making.  Even  more  dramatic  are  the  direct 
parallels  to  the  executive's  rationale  for  when  intuition  is  critically  important  and 
the  situations  faced  by  pilots  when  "things  start  to  go  wrong".  The  unprecedented, 
time  pressured,  knowledge  lean  situations  faced  by  pilots  in  emergency  situations, 
especially  chaotic,  multiple  failures  for  which  they  have  no  previous  experience  or 
training,  are  precisely  when  they  need  to  be  able  to  call  upon  and  use  the  higher 
form  of  expert  thinking  called  intuition. 

C.  Proposed  Cognitive  SOARing  Testing  and  Training 

This  section  describes  a  pilot  expert  decision  making  test  instrument  which 
emulates  and  includes  the  intuitive  assessment  aspects  of  the  Agor  Intuitive 
Management  (AIM)  Survey.  However,  it  will  go  beyond  the  identification  of  that 
specific  trait  and  test  the  cognitive  areas  (Sensing,  Organizing,  Analyzing  and 
Responding)  in  which  pilots  are  strong  and  weak  so  that  individuals,  instructors 
and  researchers  can  focus  decision  making  training  in  those  areas  needing  work. 

Part  I  of  the  SOARing  self-test  would  be  derived  by  careful  examination  of  the 
Meyers-Briggs  Type  Indicator  (MBTI)  and  selection  of  three  questions  in  each  of  the 
four  scales  to  determine  the  pilot's  basic  decision  making  type  as  done  by  the  AIM. 
Based  on  Agoi^s  success  with  executive  decision  making  testing,  the  selection  of  the 
12  questions  could  be  accomplished  jointly  by  the  contractor  and  the  FAA  Technical 
Monitor.  The  selected  questions  and  the  rationale  for  selection  could  be  validated 
with  personnel  from  the  Center  for  Applications  of  Psychological  Type  in 
Gainesville,  Florida.  This  is  a  non-profit  organization  whose  purpose  is  to  further 
research  and  education  on  the  Myers-Briggs  Type  Indicator.  Based  upon  the  results 
of  these  reviews,  the  questions  selected,  the  number  of  questions,  etc.  could  be 
revised.  Final  selection  could  also  be  reviewed  with  FAA  examiners.  Certified 
Flight  Instructors  and  Flight  Training  Schools  as  deemed  necessary. 

An  understanding  of  a  pilot's  type  in  general  and  his  own  preferences  in  responding 
to  task  demands,  will  help  define  what  individual  areas  of  EDM  require  training.  It 
will  also  indicate  what  areas  of  intrapersonal  skills  are  strong  and  weak  so  that  any 
crew  resource  management  training  can  stress  these  areas.  Finally,  it  will  form  the 
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foundation  for  closer  examination  of  how  these  preferences  are  affecting 
information  processing,  problem  solving  and  decision  making  in  Part  II  of  the  test. 

Part  II  of  the  SOARing  self-test  will  consist  of  12  flight  scenarios  selected  from  actual 
occurrences,  incidents  and  accidents  (and  again  similar  to  the  successful  ADM 
written  self-test).  The  criteria,  construction  and  development  of  these  scenarios  is 
described  in  detail  in  Appendix  F.  Each  scenario  will  be  presented  in  written  form 
followed  by  a  series  of  questions  dealing  with  Sensing,  Organizing,  Analyzing  and 
Responding  to  the  tasks  demanded  of  the  pilot  in  the  normal,  abnormal  or 
emergency  situations  described. 

Sensing  in  this  test  measures  the  pilot's  ability  to  identify  a  problem  or  an 
impending  problem  and  is  the  first  step  in  deciding  to  do  something  about  :t.  The 
critical  characteristic  that  will  be  evaluated  by  the  test  is  the  amount  of  information 
that  is  "attended  to"  without  requiring  controlled  or  focused  attention.  Since  the 

amount  of  time  pilots  spend  actively 
attending  to  sensory  inputs  vs.  passively 
monitoring  the  cues  and  context  of  a 
situation  varies  directly  with  knowledge, 
experience,  training  and  currency,  this 
scale  will  be  the  first  differentiator  in 
expert  vs.  novice  decision  making 
capabilities.  Specific  questions  in  each 
scenario  will  deal  with  subtle 
information  that  the  less  experienced 
pilots  will  have  to  spend  more  time 
"attending  to".  The  nature  of  the  questions,  and  the  timed  test  responses  will  result 
in  scores  which  are  rated  on  a  scale  of  from  1  (the  least  likely  response  to  5  the  most 
likely  response).  These  scores  will  be  accumulated  with  the  remaining  three 
decision  making  parameters  to  produce  an  assessment  on  each  individual  scale  as 
well  as  an  overall  EDM  pilot  rating. 

Organizing  involves  the  ability  to  filter,  sort  and  prioritize  sensed  information  as  a 
critical  part  in  the  understanding  of  the  situation.  The  basic  characteristics  that  will 
be  involved  in  this  part  of  the  self-test  will  be  whether  pilots  use  serial,  checklist 
processing  of  all  relevant  alternatives  or  whether  they  use  non-linear,  parallel, 
inferential  information  processing.  Again,  the  responses  will  be  scored  numerically 
from  1  to  5  and  accumulated  for  individual  EDM  self-assessments  and  overall  EDM 
ratings. 


SOAR  SELF-TEST  SCALES 
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Analyzing  and  processing  of  the  available  information  relies  on  the  type  of 
knowledge  stored  and  how  it  can  be  retrieved.  The  pilot's  style  can  vary  from  pure 
application  of  flying  skills  to  the  use  of  rules  or  heuristics  and  finally  to  the 
application  of  a  combination  knowledge,  skills  and  rules.  This  scale  will  rate  the 
pilot's  ability  to  use  large  procedural  "chunks"  and  relevant  pattern  recognition 
techniques  to  reduce  the  cognitive  searching  and  deductive  reasoning  involved  in 
processing  many  and  diverse  elements  of  conceptual  knowledge. 

Responding  is  the  most  critical  of  the  EDM  steps.  It  requires  that  the  pilot  take  some 
action  and  then  monitor  and  evaluate  the  effect  of  that  action.  The  test  scenarios 
will  evaluate  this  capability  by  offering  opportunities  for  both  selecting  one  of 
several  "alternative  solutions"  or  creating  an  individual  solution  to  the  problem 
presented.  The  scale  will  rank  the  pilots  according  to  the  degree  of  cognitive, 
associative,  autonomous  and  intuitive  ability  applied  based  both  upon  the  number 
of  alternatives  attempted  and  the  total  time  to  reach  convergence  on  the  a  solution. 

Although  a  score  will  be  associated  with  each  preference,  it  only  indicates  how 
consistently  a  pilot  chooses  one  type  of  decision  making  over  another  in  each 
category.  High  scores  generally  mean  a  clear  preference  for  expert  vs.  novice 
characteristics  in  a  specific  area.  However,  there  is  nothing  wrong  with  low  scores, 
especially  in  less  experienced  piiuls. 


EOM  RATING  METHOD 

SOAR  Ing  SELF  -  ASSESSMENT  TEST 

8  ©  A  IJ»  TOTAL 


EXPERT 


The  real  importance  of  the  profile  is  to  indicate 
a  pilots  ability  to  solve  problems  and  make 
decision  using  a  specific  type  of  decision 
making  and  problem  solving.  Expert  pilots 
need  to  know  how  to  use  the  full  range  of  their 
"thinking"  capabilities  depending  on  the  cues, 
context,  familiarity  and  difficulty  of  the 
situation.  The  important  point  is  that  research 
in  many  fields  has  shown  that  this  is  a  skill 
which  can  be  acquired  through  practice  and 
training.  That  is  the  ultimate  purpose  of  this 
self-test  exercise.  The  continued  development  of  these  testing  and  training 
materials  is  proposed  as  Phase  III  of  the  current  FAA  research. 
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6.0  CONCLUSIONS  and  RECOMMENDATIONS 

This  section  summarizes  what  pilots  need  to  know  to  use  the  full  range  of  their 
"thinking"'  capabilities.  The  expert  pilot's  decision  making  "adapts"  depending  on 
the  cues,  context,  familiarity  and  difficulty  of  the  situation.  Research  in  many  fields 
has  shown  that  at  least  some  of  the  expert  decision  making  skills  can  be  acquired 
through  practice  and  training.  The  ultimate  purpose  of  this  report  is  to  suggest  a 
starting  point  for  developing  EDM  training  for  pilots.  The  following  conclusions 
and  recommendations  are  tailored  to  that  purpose. 

CONCLUSION  1:  Enhanced  memory  and  problem  solving  training  are  required  to 
expedite  development  of  Expert  Decision  Making  (EDM)  in  novice  pilots. 

The  study  of  cognition  has  identified  the  importance  of  sensory,  short  term  and  long 
term  memory  in  the  development  of  expertise.  Since  memory  is  the  fundamental 
process  that  accounts  for  the  differences  in  expert  and  novice  decision  making,  the 
use  of  these  systems  in  information  processing  and  problem  solving  is  critical  to 
training  expert  pilot  decision  making.  Attention,  perception,  pattern  recognition, 
associations,  elaborations,  inferences  and  how  basic  conceptual  knowledge  is  stored 
all  impact  recall  and  the  level  of  automaticity  in  decision  making.  Developing  these 
characteristics  into  an  integrated  set  of  cognitive  procedural  knowledge  is  required 
to  progress  to  the  next  stage  of  decision  making  training. 

RECOMMENDATION  1  -  Aviation  problem  solving  and  pilot  memory  training 
courses  should  be  developed.  Three  non-aviation  cognitive  training  programs 
which  offer  potential  insight  for  new  EDM  training  paradigms  were  reviewed  in 
Section  5.2.  These  and  other  programs  which  have  documented  improvements  in 
attention,  memory  and  problem  solving  should  be  explored  and  adapted  to 
prototype  EDM  courseware. 

CONCLUSION  2:  The  reliance  of  EDM  on  knowledge  based  on  experiences  and 
sensory  stimuli  make  the  selection  of  training  media  extremely  critical. 

This  study  has  shown  that  experts  structure  information  differently  in  memory  and 
integrate  new  information  rapidly.  Development  of  EDM  is  influenced  by  the  cues 
and  task  demands  encountered  in  the  course  of  experience  and  training.  Expert 
pilots  change  a  structured,  deductive  process  into  a  cued  recall  tisk  by  building 
mental  representations  of  meaningful  relationships  in  a  situation.  These 
relationships  are  more  than  the  cognitive  knowledge  perceived  by  novices  in  the 
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same  situation.  They  provide  keen,  quick  insight  and  almost  a  direct  perception  of 
the  proper  course  of  action.  Expert  decision  making  in  new  situations  with 
ambiguous  information  and  rapidly  changing  cues  requires  an  adept  use  of  this 
process.  The  expert's  perceptual  superiority,  fast  access,  adaptive  decision  making 
requires  broad  exposure  to  meaningful  sensory  cues  in  operational  contexts  to 
develop  this  inferential  capability. 

RECOMMENDATION  2  -  The  development  of  a  knowledge  based  CBT  or  IVT 
expert  training  device  should  be  initiated.  The  fields  of  Activity  Based  Learning 
(ABL),  Computer  Based  Training  (CBT)  and  Interactive  Video  Training  (IVT)  offer 
near  term,  cost  effective  alternatives  which  are  attractive  to  both  air  lines  and 
general  aviation  pilots.  Two  example  CBT  systems  were  discussed  in  Section  5.2. 

CONCLUSION  3:  Experience  since  1986  with  ADM  training  in  both  civil  and 
military  U.  S.  operations  has  shown  that  decision  making  can  be  taught  and  this 
training  reduces  the  number  of  human  error  accidents. 

Prior  to  the  development  of  ADM  training,  it  was  widely  held  that  good  decision 
making  could  only  be  learned  through  experience.  The  15  years  of  ADM  work  has 
contradicted  this  logic  by  showing  that  some  aspects  of  good  decisions  can  be  trained. 
However,  we  have  now  realized  that  there  is  another,  more  subtle  impact  of 
experience  on  decision  making  in  which  the  steps  of  the  process  are  hidden  in  the 
perceptual  and  procedural  portion  of  human  information  processing.  The  expert's 
superior  perceptual  ability  is  aided  by  the  ability  to  store  and  access  chunks  of  data  in 
short  term  memory.  The  superior  problem  solving  ability  is  speeded  up  by  the 
storing  of  patterns  or  schema  of  related  events,  facts  and  results  in  long  term 
memory.  Experts  also  develop  superior  situational  awareness  capabilities  which 
including  self  regulatory  processes  that  free  working  memory  for  higher  levels  of 
conscious  processing.  These  abilities  include:  planing  ahead,  efficiently  monitoring 
one's  time  and  attention  resources  and  monitoring  and  editing  one's  efforts  to  solve 
a  problem. 

RECOMMENDATION  3  —  A  second  generation  ADM  or  EDM  training  manual 
should  be  developed  in  response  to  industry's  request  for  additional  material  and  to 
continue  the  safety  improvements  generated  by  this  training.  This  manual  should 
focus  on  the  superior  expert  memory,  information  processing  and  problem  solving 
characteristics  identified  in  this  report.  Much  of  the  material  needed  for  this 
manual  has  already  been  created  during  Phases  I  and  II  of  this  project.  Reformatting 
the  EDM  research  into  a  training  manual  could  be  accomplished  in  a  timely,  cost 
effective  manner. 
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CONCLUSION  4:  An  FAA  EDM  training  program  should  be  developed  and  tested. 
Courseware,  an  instructor's  syllabus,  student  manuals  and  self  test  materials  are 
required. 

The  successful  use  of  ADM  training  material  in  the  U.S.  and  internationally  has 
shown  that  the  quality  of  decision  making  can  be  enhanced  with  the  appropriate 
development  of  cognitive  skills  in  a  classroom  setting.  Based  on  the  research  from 
Phase  I  and  Phase  II  of  this  project,  it  appears  that  metacognition  training  (under 
evaluation  using  novice  pilots  at  Massey  University  in  New  Zealand)  and  intuitive 
decision  making  training  (i.e.,  training  based  on  experiences  and  acquired  through 
sensory  contact  offer  significant  potential  for  further  reductions  in  pilot  error 
accidents  related  to  decision  making.  These  include  providing  the  novice  pilots  the 
knowledge  and  procedures  necessary  to  assess,  interpret  and  understand  their  own 
"thinking"  or  decision  making  processes.  The  high  level  information  processing 
skills  being  taught  at  Massey  include  the  individuals  approach  to  decision  making, 
self-assessment  and  metacognition  skills  in  a  time-pressured,  dynamic  decision 
making  situation. 

RECOMMENDATION  4  --  The  aviation  EDM  training/testing  conducted  at  Massey 
University  in  New  Zealand  since  1990  should  be  the  starting  point  for  an  FAA  EDM 
course  for  pilots.  The  Intuitive  Decision  Making  testing  from  non-aviation  fields 
should  be  reviewed  for  possible  relevance  or  enhancements  of  the  Massey  program. 


CONCLUSION  5:  A  written  self-assessment  test  would  be  the  most  expeditious 
and  cost  effective  method  of  raising  the  awareness  of  aviator  EDM  characteristics 
and  initiating  EDM  training. 


SOAR  SELF-TEST  SCALES 


The  application  of  the  knowledge  of 
expertise  to  the  field  of  Aeronautical 
Decision  Making  identified  four  aspects  as 
critical  to  Expert  Decision  Making.  These 
were:  Sensing,  Organizing,  Analyzing  and 
Responding  to  the  cues,  context  and 
demands  of  the  situation.  These  four 
mental  aspects  are  integral  to  the 
preparation,  analysis  and  execution  of 
decisions  for  normal,  emergency,  and, 
abnormal  or  novel  situations.  They  also  can  form  the  basis  of  both  a  testing  and 
training  program  for  EDM.  The  type  of  problem  solving  behavior  where  solutions 
are  developed  under  time  pressured,  ambiguous  circumstances  has  been  identified 
as  Type  C  problem  solving  behavior.  Successful  training  of  this  behavior  has  been 
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demonstrated  in  other  fields.  The  transfer  of  this  training  to  aviation  was  initiated 
during  Phase  II  of  this  study  and  was  termed  Cognitive  SOARing. 

RECOMMENDATION  5  --  Continue  development  of  the  Cognitive  SOARing  self- 
assessment  test  initiated  for  assessing  individual  pilot  EDM  capabilities.  Integrate 
this  self-assessment  procedure  into  both  the  future  EDM  training  manuals  and 
course  material  developed. 

CONCLUSION  6:  Selecting  realistic  EDM  scenarios  with  critical  events  and 
performing  a  cognitive  task  analysis  of  novice  vs.  expert  decision  making  for  these 
events  is  a  basic  requirement  for  future  EDM  training  development. 

The  central  problem  when  studying  EDM  in  aviation  is  that  the  process  to  be 
studied  occurs  in  complex  and  highly  interactive  situations.  The  general  solution  to 
this  problem  (developed  in  other  areas  of  expertise)  is  to  analyze  carefully  the  real 
life  events  requiring  EDM  and  to  identify  brief  segments  of  the  expert's  behavior 
that  corresponds  to  a  natural  occurring  task.  This  type  of  cognitive  task  analysis  was 
suggested  as  the  next  significant  step  in  the  development  of  decision  making 
training  by  the  participants  in  a  1992  ADM  workshop  (Adams  &  Adams,  1992).  In 
many  task  analyses  it  is  possible  to  explicate  a  limited  number  of  sequences  of 
thoughts  that  will  adequately  describe  how  subjects  generate  the  best  answer  or 
some  alternative  answer  for  a  given  task,  these  alternative  thought  sequences  make 
it  feasible  not  only  to  describe  other  subjects'  efforts  to  complete  the  task,  but  would 
provide  the  potential  of  giving  feedback  to  subjects  generating  sub-optimal 
answers/ actions.  Analyses  of  expert  performance  using  this  type  of  task  analysis  has 
revealed  a  perspective  of  general  characteristics  of  experts'  cognitive  processes. 

RECOMMENDATION  6  —  The  design  of  EDM  training  scenarios  and  assessment 
methods  should  be  initiated  in  parallel  with  training  materials,  training  tools  and 
courseware  development.  A  methodology  and  implications  of  developing  EDM 
scenarios  are  presented  in  Appendix  F.  This  type  of  methodology  is  required  for 
Activity  Based  Learning  (Recommendation  2),  a  second  generation  training  manual 
(Recommendation  3)  and  self-assessment  written  testing  (Recommendation  5). 
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APPENDIX  A 


EXPERT  INFORMATION  ANALYSIS  AND  EDM  DEVELOPMENT 

Cognitive  psychology  is  the  study  of  how  people  think,  Cognition  in  its  simplest 
definition  is  "thinking".  Processing  is  the  mental  function  that  takes  in  sensory 

inputs  and  transforms  them  into  information  that 
can  be  used  in  thinking.  Sensory  inputs  include 
visual,  aural  and  kinesthetic  (vibration,  acceleration, 
etc.  )  data  from  the  environment.  Processing 
involves  sampling  data  from  the  situation  and 
identifying  it  as  normal  or  abnormal.  Decision 
making  includes  anticipating  problems  or  problem 
solving  if  one  already  exists. 

Captain  A1  Haynes'  information  processing  abilities 
were  discussed  in  Section  3.1  as  one  example  of 
cognitive  SOARing:  Sensing,  Organizing,  Analyzing  and  Responding  to  a  highly 
stressful  and  time  critical  decision  making  circumstance.  However,  pilots  and  crews 
use  these  four  steps  on  a  daily  basis  for  both  normal  and  emergency  problem  solving 
and  decision  making.  The  following  discussion  addresses  the  unique  mental 
abilities  that  allow  expert  pilots  to  be  successful  in  routinely  executing  these  steps. 
The  purpose  of  this  discussion  is  to  identify  the  basic  elements  and  concepts  which 
heighten  a  pilot's  information  processing  abilities. 

A.l  Information  Processing  Systems 

The  stud)  of  cognition  over  the  past  40  years  has 
identified  he  import. "ice  of  four  systems  used  m 
thinking  or  "inform  m  processing".  1  m  -.tr 
system^  ore  illustrated  m  the  Human  Information 
Processing  U  IIP)  di-igram. 

Basically,  these  systems  include:  the  sensory  systems 
(visual,  auditors,,  se.il-of-the-pants,  etc.),  the  memory 
systems  (long  term,  short  term  and  sensory  memory),  the  processor  and  the 
response  systems  (motor  events,  commun  .  tions,  etc.). 

MEMORY  Human  memory  is  the  fundamental 
process  that  accounts  for  the  differences  in  expeii 
and  novice  pilot  thinking  capabilities.  Central  to 
that  process  is  the  way  information  is  perceived  and 
stored  or  organized.  Perception  relies  on  the 
abilities  of  the  sensory  memory  to  recognize  and 
sort  information.  Organizing  information  requires 
the  resources  of  the  short-term  and  long  term 
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memory  systems.  Both  types  of  memory'  are  critically  impacted  by  the  quantity  and 
type  of  "knowledge  available.  In  fact,  the  topics  of  type  of  knowledge,  how  it  is 
represented  in  memory,  how  people  tie  it  together  and  how  they  remember  it  are 
central  to  the  understanding  of  cognition  in  general  and  how  pilots  think  in 
particular. 

Sensory  memory  insures  that  enough  retention  to  allow  sufficient  spare  processing 
capacity  is  available  for  "handling"  new  inputs.  Visual  or  iconic  memory  is 
available  for  0.5  to  1.0  seconds  and  auditory  or  echoic  memory  for  2.0  to  8.0  seconds. 
Sensing  of  information  is  a  function  of  knowledge,  attention,  perception  and 
expectations.  For  example,  the  casual  mention  of  your  name  in  the  noisy 
environment  of  a  reception  or  cocktail  party  will  quite  often  "pull"  you  into  a 
conversation  you  were  not  even  aware  of  prior  to  hearing  your  name.  This 
"cocktail  party  effect"  illustrates  the  importance  of  automatic  attention  to  your 
perception  of  contextual  cues  of  any  situation.  Expectations  also  impact  your 
sensing  powers.  Since  each  pilot  has  a  unique  background  of  knowledge,  training 
and  experience,  expectations  based  on  that  background  impact  the  ability  to  respond 
in  a  timely  fashion  to  specific  situations. 
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The  basic  characteristic  of  sensory  memory  that  is 
important  for  pilots  to  be  aware  of  is  that  a  lot  of 
HUP  -  SENSORY  MEMORY  information  is  "sensed"  or  received,  but  only  a 

Aiotoi  iniwmauon  received,  •  •man  amount  small  amount  is  "attended  to"  or  remembered.  An 

. H'Y.?*,!!.0! *od  - example  of  this  phenomena  is  the  large  amount  of 

<focu»W)  digital  and  analog  information  sampled  or  sensed 

ansmoN  by  pilots  in  a  typical  scan  of  the  instruments.  Most 

of  the  needle,  ball  and  airspeed  information  (if 
PERCEPTION  <r:z  within  limits)  is  simply  observed  and  noted  in 

(owfc-)  passing.  However,  if  crosstrack  deviation  or 

altitude  are  outside  desired  values,  the  pilot  is  generally  quick  to  notice  and  make 
appropriate  corrections.  This  is  an  example  of  controlled  or  focused  attention  which 
involves  a  conscious  decision  to  respond  and  usually  occupies  sensory  memory  on  a 
single  task  at  any  given  time. 

On  the  other  hand,  while  scanning  the  instruments,  listening  to  the  radio  and 
developing  a  mental  picture  of  the  flight's  progress,  a  pilot  will  easily  "pick-up"  his 
call-sign  when  it  is  uttered  by  ATC  and  immediately  respond.  This  is  an  example  of 
passive  attention  which  does  not  require  dedicated  sensory  memory  until 
stimulated  by  salient  cues.  The  amount  of  time  pilots  spend  actively  attending  to 
sensory  inputs  vs.  passively  monitoring  the  cues  of  a  situation  varies  directly  with 
knowledge,  experience,  training  and  currency.  Dedicated  time  spent  focusing  on 
individual  cues  and  responding  is  time  taken  away  from  situation  monitoring  or 
passive  situation  assessment.  For  these  reasons,  "Attention"  is  one  of  the 
differentiators  which  can  be  used  to  identify  experts  vs.  novices.  Attention  can  also 
provide  the  first  part  of  a  training  program  designed  to  lessen  the  reliance  on 
experience  as  the  only  means  of  attaining  expert  performance. 


Short-term  memory  Short  and  long  term  memory  should  not  be  thought  of  as 
different  places  pilots  "put"  facts  or  different  types  of  information.  Rather,  the 
difference  in  these  two  memory  systems  is  based  in  the  level  of  activation  of  the 
knowledge  at  any  given  time.  The  Short  term  or  Working  memory  system  consists 
of  a  limited  amount  of  information  that  is  in  a  special  "active"  state  available  for 
immediate  use.  Working  memory  has  two  distinctive  characteristics.  It  is  limited 

in  both  capacity  and  time.  A  typical  20  year  old 
student  pilot  will  be  able  to  remember  from  five  to 
nine  items  for  a  transient  period  of  from  18-20 
seconds. 


As  pilots  get  older,  both  the  number  of  items  in 
Working  memory  and  the  duration  of  their  active 
state  decrease.  However,  this  is  not  an  all  bad  news 
story  since,  once  again,  experience  and  training  can 
have  a  major  impact  on  how  much  useful 
information  each  "item"  contains.  In  fact,  experts 
in  all  fields  develop  the  ability  to  store  "chunks"  of  related  data  which  novices 
might  be  forced  to  store  individually  to  retain  their  active  state  (see  the  chart  for 
examples  of  chunks).  For  example,  the  expert  pilot  tuning  in  the  ATIS  broadcast 
prior  to  entering  a  terminal  area  knows  to  expect:  time,  ceiling,  visibility, 
temperature,  dew  point,  wind,  altimeter  and  runway-in-use  information  and  will 
not  be  overwhelmed  with  the  task  of  "remembering"  the  chunks  that  are 
transmitted.  Conversely,  a  novice  may  have  to  write  down  the  significant  ceiling, 
visibility,  wind,  altimeter  and  runway  data. 

In  summary,  the  short  term  memory  provides  active,  usable  chunks  of  information 
in  a  state  of  readiness  to  be  used.  The  precise  content,  organization  and  use  ability  of 
each  chunk  is  tied  directly  to  exposure  and  practice.  Therefore,  pilot  training  and 
especially  decision  making  training  could  benefit  by  re-examining  the  criticality  of 
information  chunking  and  train  novice  pilots  earlier  in  the  necessary  short  term 
memory  skills. 
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Long  term  memory  The  long  term  memory 
system  is  not  simply  an  encyclopedic  repository  of 
facts.  Rather,  it  is  stored  knowledge  (including 
factual,  procedural,  experiential  and  emotional) 
that  individuals  have  stored  in  related  groups  and 
have  the  ability  to  reactivate.  Reactivation  can  be 
initiated  by  the  cues  (mental  or  physical),  the 
context  of  a  situation  (standard  procedure  or 
emergency)  and  the  pilot's  abilities  to  make 
associations,  retrieve  and  activate  the  needed 
knowledge.  Whereas  the  chunks  of  information 
in  short  term  memory  are  active  by  definition,  the 
level  of  activation  of  the  knowledge  in  long  term 
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memory  is  based  upon  the  recency  with  which  it  has  been  learned  and  experienced 
and  the  number  of  times  it  has  been  used  (or  practiced)  in  new  situations. 

Long  term  memory,  then,  depends  on  the  pilot's  ability  to  respond  to  new  demands 
for  information  through  his  abilities  of  recognition  and  recall.  Recognition  powers 
allow  the  pilot  to  discriminate  between  the  new  situation  and  similar  or  different 
"memories"  which  might  apply.  Recall  allows  the  pilot  to  not  only  discriminate, 
sort  and  prioritize  the  information  that  is  needed,  but  also  aids  in  the  generation  of 
new  ideas,  solutions  or  actions  required  to  respond  to  the  demands  of  situations 
previously  not  experienced. 

Once  again,  expert  pilots  (as  well  as  experts  in  other  fields)  have  developed  superior 
long  term  memory  organizational  capabilities  which  facilitate  recognition  and  recall 
of  their  knowledge.  Experts  use  schema  (a  modifiable  information  structure  or 
concept  based  upon  knowledge  that  is  experienced)  to  recognize  and  interpret  the 
cues  and  context  of  a  new  situation  based  upon  their  related  knowledge.  That  is, 
experts  experience  and  store  interrelationships  between  objects,  data,  situations  and 
results.  This  schema  theory  states  that  experts  store  schemata  for  recurrent 
situations  that  expedite  decision  making  and  problem  solving.  The  simplest 
example  of  this  ability  in  aviation  is  the  expert  pilots  superior  knowledge  of 
procedures  for  both  normal  and  emergency  situations.  More  complex  examples 
would  include  the  experienced  pilot  anticipating  entering  a  thunderstorm, 
anticipating  wind  shear  on  landing  or  anticipating  in-flight  icing  conditions.  These 
abilities  are  also  manifested  by  the  expert  pilot's  traits  of  superior  situational 
awareness  (i.e.,  all  elements  including:  the  aircraft,  the  environment,  self,  etc.). 

Furthermore,  the  capability  of  experts  to  fast-access  their  knowledge  facilitates 
perception  and  leads  to  the  reduction  of  the  role  of  memory  search  and  general 
processing.  Although  the  novice  and  expert  have  equal  capability  for  cognitive 
processing,  novices  typically  use  lots  of  search  and  processing  in  a  less  focused,  more 
general  manner.  The  outstanding  performance  of  experts  is  derived  from  how  their 
knowledge  is  structured  for:  Retrieval,  Pattern  Recognition  and  Inference. 

This  expert  capability  manifests  itself  in  the  ability  to  intuitively  respond  to  patterns 
without  decomposing  them  into  component  features  or  problem  elements.  An 
example  of  these  abilities  would  be  the  expert's  ability  to  respond  to  loss  of  an  engine 
on  takeoff  without  consciously  "thinking  through"  the  engine  out  procedure.  This 
understanding  occurs  effortlessly  due  to  discriminations  resulting  from  prior, 
concrete  experience. 

In  summary,  expert  pilot's  have  tailored  their  use  of  long  term  memory  by  storing 
organized  patterns  of  related  facts,  events  and  feelings  which  affect  both  recognition 
and  recall.  Their  ability  to  fast  access  patterns  of  aviation  knowledge  is  expedited  by 
the  associations  with  cues  and  context  of  new  situations  which  stimulate  the  recali 
process.  Although  the  associations  (or  concept  elaborations)  are  predominantly 
based  upon  experience  today,  training  aimed  at  replicating  this  ability  is  not  an 
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A.2  EDM  DEVELOPMENT 

From  a  cognitive  psychology  perspective,  the  process  of  developing  more 
sophisticated  decision  making  involves  three  stages  of  expert  problem  solving  skills 
(Anderson,  1985).  These  are  cognitive,  associative  and  autonomous  .  During  the 
first,  cognitive  stage,  novice  pilots  commit  to  memory  a  set  of  facts  relevant  to  a 

desired  skill.  They 
typically  rehearse 
these  facts  as  they  first 
perform  the  skill.  For 
example,  novice 
pilots  learning  stall 
recovery  will 
memorize:  recognize 
the  stall,  lower  the 
nose,  apply  full 
power,  level  the 
wings  and  minimize 
altitude  loss.  In  this 
stage,  they  are  using 
their  general 
aeronautics 

knowledge  to  guide  their  solution  to  loss  of  lift  over  one  wing,  and  solve  a  specific 
problem,  how  to  keep  the  aircraft  flying.  The  problem  solving  capabilities  and  level 
of  expertise  in  this  stage  are  very  basic.  These  skills  rely  principally  on  serial 
(checklist)  type  problem  solving,  a  lengthy  general  search  of  conceptual  knowledge 
and  controlled  attention  throughout  the  resolution  of  the  problem.  As  the 
experience  base  increases,  the  cognitive  decision  maker  develops  heuristic  decision 
making  capabilities  for  those  learned  problems  that  have  been  practiced  or 
encountered. 
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unreasonable  g^al.  In  addition  to  the  elaborations, 
the  expert's  ability  to  use  inferences  to  aid 
reconstruction  of  similar  problems  and  solutions 
could  also  comprise  part  of  this  training.  The  final 
important  element  of  such  a  training  program 
would  be  to  teach  the  novice  to  go  back  to  th<*  cues 
and  context  of  the  current  problem  to  affirm  that  his 
response  is  the  correct  one. 


The  second,  or  associative  stage,  has  two  important  characteristics.  First,  errors  in 
the  initial  understanding  and  performance  are  detected  and  gradually  eliminated. 
The  novice  pilot  learns  to  coordinate  the  nose  drop,  power  application  and  rudder 
application  for  a  smooth  stall  recovery.  Second,  the  connections  between  the 
various  elements  required  for  successful  performance  are  strengthened.  The  pilot 
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does  not  sit  for  a  few  seconds  trying  to  decide  which  action  to  perform  first  after 
lowering  the  nose.  Basically,  the  outcome  of  the  associative  stage  is  a  learned 
procedure  or  production  rule  for  performing  a  desired  response  to  a  known 
situation.  In  this  stage,  the  conceptual  knowledge  is  transformed  into  a  procedural 
form  which  provides  the  pilot  with  the  capabilities  for  more  dynamic  thinking  and 
parallel  processing.  However,  the  procedural  form  does  not  necessarily  replace  the 
conceptual  knowledge.  Rather  the  two  forms  coexist  and  are  available  when  needed 
for  a  task.  For  example,  the  low  time  pilot  can  fly  the  airplane  while  simultaneously 
talking  to  ATC  and  navigating.  All  the  while,  he  still  remembers  the  rules  of 
aerodynamics,  the  characteristics  of  a  stall  and  the  recovery  process. 

The  third  cognitive  stage  occurs  when  the  problem  solving  procedures  become 
faster  and  more  automated.  There  is  not  necessarily  any  sharp  distinction  between 
the  associative  and  autonomous  stages  of  expertise.  Rather,  the  autonomous  stage 
evolves  from  the  repeated  application  of  known  patterns  and  their  associative  use 
to  achieve  solutions  The  use  of  conceptual  knowledge  often  disappears  during  this 
stage,  at  least  for  some  tasks.  In  fact,  the  ability  to  verbalize  knowledge  of  the 
procedure  can  be  lost.  Furthermore,  expert  decision  making  appears  to  develop 
continually  in  a  specific  area  like  aviation.  Throughout  the  development,  the  skill 
gradually  improves. 


Ultimately,  the  skill  can  be  extended  to  the  ability  to  respond  to  cues  not  previously 
encountered  and  to  develop  new,  creative  (opportunistic)  solutions  applicable  to 

previously  unencountered  (novel 
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or  chaotic)  situations.  In  general, 
expert  decision  makers  have  the 
ability  to  adapt  to  the  inherent 
demands  and  constraints  of  a  task. 

If  the  task  can  be  done  most 
efficiently  with  forward  reasoning 
and  the  development  of  new 
production  rules  based  upon 
procedural  knowledge,  the  expert 
will  search  forward;  if  backward 
search  of  conceptual  knowledge  is 
better,  the  expert  searches  backward. 
If  certain  patterns  of  cues  are  crucial 
to  performing  the  task  well,  the 
expert  will  be  apt  to  perceive  and 


remember  them;  if  patterns  are  not 
so  important,  the  expert  will  not 

selectively  process  them.  Most  importantly,  the  expert  decision  making  capabilities 
occur  so  fast  that  they  appear  to  take  on  the  characteristics  of  insight  or  intuition. 
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APPENDIX  B 


CHARACTERISTICS  OF  INTUITION 

The  status  of  current  work  on  intuition  is  thoroughly  summarized  in  '  Intuition: 
How  We  Think  and  Act"  (Bastic,  1982).  This  is  an  exhaustive  literature  search  of 
2,692,000  articles,  reports,  and  theses  from  five  data  bases.  Only  91  had  the  word 
"intuition"  in  the  title  or  description.  Of  these  24  were  useful  studies  of  intuition 
The  remainder  only  referred  casually  to  the  word  intuition.  From  this  extensive 
review,  Bastic  compiled  20  summary  properties  of  intuition.  The  properties  are 
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j  INTUITION  PROPERTIES 

1 .  QUICK,  IMMEDIATE,  APPEARANCE 

2.  EMOTIONAL  INVOLVEMENT 

3.  PRECONSCIOUS  PROCESS 

4.  NOT  ANALYTICAL  PROBLEM  SOLVING, 
LOGIC  OR  ABSTRACT  REASONING 

5.  INFLUENCED  BY  EXPERIENCE 

6.  EMOTIVE  NOT  TACTILE 

7.  ASSOCIATED  WITH  CREATIVITY 

8 .  ASSOCIATED  WIT!  I EGOCENTRICITY 

9.  INTUITION  NEED  NOT  BE  CORRECT 

10.  SUBJECTIVE  CERTAINTY 
OF  CORRECTNESS 

1 1 .  RECENTERING 

1 2 .  EMPATHY.  KINESTI IETIC  OR  OTHER 

1 3 .  INNATE,  KNOWLEDGE  OR  ABILITY 

1 4 .  PREVERBAL  CONCEPT 

15.  GLOBAL  KNOWLEDGE 

16.  INCOMPLETE  KNOWLEDGE 

1 7 .  HYPNOGOGIC  REVERIE 
18. SENSE  OF  RELATIONS 

19.  DEPENDENCE  ON  ENVIRONMENT 

20 .  TRANSFER  AND  TRANSPOSITION 
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KNOWLEDGE  LEAN  PROBLEM  SOLVING 

PATTERN  RECOGNITION 

CUES  AND  CONTEXT  OF  SITUATIONS  OR 

DECISION  TASK  DEMANDS 


repeated  here  with  those  properties  most  applicable  to  how  expert  pilots  think 
indicated  in  bold  type.  These  properties  are  presented  as  a  set  of  "descriptors"  of 
intuition,  but  are  not  independent.  Rather,  they  are  related  in  many,  complex  ways 
some  of  which  will  be  explained  in  the  text  which  follows.  Where  there  is  an 
apparent  impact  on  Expert  Decision  Making  and  an  opportunity  for  EDM  training, 
this  will  also  be  described. 
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PROPERTIES  1  and  2  --  These  properties  reflect  the  immediate  knowing  or 
perception  of  a  solution  to  a  problem,  or  what  action  to  take  in  a  situation,  without 
the  conscious  use  of  reasoning.  This  "instantaneous  apprehension"  is  the  Eureka 
experience  we  are  all  familiar  with.  The  closest  scientific  term  that  describes  this 
experience  is  insight  which  Webster's  defines  as  "a  clear  understanding  of  the  inner 
nature  of  some  specific  thing".  From  an  operational  aviation  perspective,  we  will 
use  intuition  and  insight  as  interchangeable. 

PROPERTY  3  —  Preconscious  is  used  to  refer  to  material,  which,  though  at  the 
moment  it  may  be  unconscious,  is  available  and  ready  to  be  consciously  used  in 
decision  making  (Drever,  1974).  Preconscious  process  refers  to  two  properties  of 
intuition:  preconscious  information  intake,  and  preconscious  intuitive  processing 
of  this  information.  The  preconscious  processing  property  underlies  the  EDM 
capability  for  autonomous  problem  solving.  In  this  sense,  preconscious  means 
direct  knowledge  of  an  empirically  derived  decision  without  conscious,  analytical 
awareness  of  its  basis.  This  preconscious  property  is,  however,  based  upon 
principled  knowledge  such  as  aerodynamics,  gravity,  Newton's  Law,  etc. 

PROPERTY  4  --  Intuition  contrasts  with  abstract  reasoning,  logic  and  analytical 
thought.  Although  there  is  evidence  that  shows  cither  an  intuitive  or  abstract, 
analytical  approach  to  problem  solving  is  a  stable  cognitive  style,  there  are  basic 
observable  differences.  The  analytical  process  of  decision  making  is  a  step-by-step 
process  which  normally  just  compares  two  elements  at  a  time.  Intuitive  processing 
is  based  on  an  overall  impression  or  a  knowledge  state  that  associates  all  the 
elements  of  a  specific  field.  Pilot  intuition  uses  parallel  processing  based  upon 
associations  and  non-linear  feedback  from  the  whole  field  of  aviation  knowledge, 
simultaneously.  This  process  accounts  for  the  dramatic  difference  in  the  speed  of 
processing.  Since  the  concurrent  feelings  associated  with  the  situation  have  been 
associated  in  memory  by  the  pilot's  unique  experience,  original  connections  or 
associations  can  result.  However,  these  elements  are  related  by  a  common 
subjective  feeling,  not  by  logic. 

PROPERTY  5  --  Intuitive  decision  making  is  dependent  on  the  past  experiences  and 
the  present  situation  of  the  pilot.  Stimuli  (cues)  from  the  environment  and  goals 
(context)  of  the  situation  are  used  in  memory  to  "perceive"  the  characteristics  of  the 
"whole"  problem.  The  experience  based  "perception"  includes  the  pilot's  motor, 
emotional  and  cognitive  cues  as  a  part  of  the  problem  categorization  and  decision 
making  process.  Past  experience  can  also  interfere  with  perceptions  of  a  situation 
when  anxiety,  stress,  emotional  blocking  or  cognitive  dissonance  associations  are 
triggered.  What  is  intended  by  the  use  of  intuition  in  the  decision  making  process  is 
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a  perception  of  the  essential  requirements  of  a  problem  so  that  past  experience  can 
be  applied  in  a  manner  appropriate  to  the  present  situation.  The  full  implications  of 
the  dependency  of  expert  decision  making  training  on  the  ability  to  replicate  this 
"whole"  experience  may  be  crucial  to  future  training  development. 

PROPERTIES  6,  9  AND  12  --  These  properties  indicate  that  intuition  is  a  sense  of 
feeling  for  patterns  or  relationships.  The  subjective  associations  of  feelings  between 
the  elements  of  a  situation  seem  to  hold  the  elements  in  a  pattern  or  schema  for  use 
in  later  processing.  These  insights  are  not  necessarily  correct  due  to:  the  emotional 
involvement;  the  lack  of  total  representation  of  the  whole  problem;  or,  empathy  of 
the  pilot  with  the  objects  or  events  of  the  situation. 

PROPERTY  7  --  The  creativity  involved  with  intuition  is  essential  to  adaptive  expert 
decision  making.  The  pilot  faced  with  a  novel  situation  (one  for  which  he  has  not 
been  trained  or  has  not  experienced)  — st  be  able  to  combine  his  knowledge, 
training  and  experience  derived  patterns  in  a  timely,  strategic  manner.  This  has 
been  referred  to  as  "opportunistic  planning"  in  the  EDM  development  discussion. 
However,  these  plans  derived  irom  preconscious  mental  activity  may  not  always  be 
correct.  Therefore,  in  aviation,  mathematics  and  scientific  fields,  the  creative 
decision  is  always  followed  by  logical,  empirical  verification  of  the  intuition. 

PROPERTY  8  --  The  egocentricity  property  of  intuition  simply  means  that  situations 
that  prove  embarrassing  or  in  which  the  pilot  is  the  center  of  attention  appear  to  be 
remembered  better  than  less  "ego-involved"  experiences.  Pure  egocentricity  is  lack 
of  the  ability  to  differentiate  self  interest  from  the  perceived  environment  or 
situation.  The  extreme  result  of  this  property  would  be  intuitive  thought 
dominated  by  preoccupation  with  personal  satisfaction. 

PROPERTY  10  --  Irrespective  of  the  objective  correctness  of  the  intuition,  it  is 
accompanied  by  a  feeling  of  confidence  related  to  the  relief  of  anxiety  and  tension 
associated  with  solving  an  ill-structured  problem.  This  subjective  certainty  has  been 
documented  in  several  aviation  "saves"  where  the  pilot  quickly  responded  to 
multiple  emergencies  with  confidence  that  he  would  safely  land  the  airplane  but 
without  any  logical  or  objective  reasons  for  that  confidence.  This  confidence  or 
optimism  based  upon  unsubstantiated  evidence  is  often  an  accurate  predictor  of 
correct  insights  or  intuitions. 

PROPERTY  11  --  Recentering  is  sometimes  called  "cognitive  reorganization" 

(Watson,  1974).  It  is  the  experience  that  suddenly  something  changes  in  a  situation 
or  problem  into  one  that  is  familiar  and  can  be  dealt  with  when,  prior  to 
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recentering,  there  was  a  feeling  of  frustration  and  inability  to  cope.  This  term  has 
also  been  applied  to  solutions  or  ideas  that  appear  "out-of-the-blue"  after  the 
problem  has  been  mentally  shelved  or  the  pilot  is  involved  with  some  activity  that 
is  apparently  unrelated  to  the  original  problem. 

PROPERTIES  13  &c  16  --  Intuitive  knowledge  is  sometimes  considered  to  be  innate  or 
instinctive.  This  type  of  knowledge  is  used  to  arrive  at  judgments  or  solutions  for 
which  the  pilots  knowledge  is  too  incomplete  to  justify  a  decision.  "Woman's 
intuition"  falls  in  this  category.  This  property  also  falls  under  the  definition  of 
"primary  intuition".  This  discussion  of  the  role  of  intuition  in  developing  EDM 
does  not  include  innate  or  instinctive  knowledge  or  ability. 

PROPERTY  14  --  This  property  of  intuition,  Preverbal,  is  used  for  two  reasons.  First, 
because  intuitive  perception  and  processing  is  speedy  on  many  parallel  associations 
where  the  verbal  mode  of  processing  is  linear  and  slow  in  comparison.  This  speedy 
parallel  vs.  slow  linear  comparison  is  also  a  reason  behind  the  contrast  of  intuition 
with  logic.  Second,  the  amount  and  quality  of  knowledge  which  the  pilot  receives 
through  intuition  makes  its  complete  verbalization  impossible.  This  property  is  not 
critical  to  the  EDM  training  objectives  of  this  project. 

PROPERTY  15  &  18  -  The  essence  of  intuition  and  autonomous  cognitive 
processing  is  in  the  wholeness  of  perception,  the  completeness  of  ideas,  the 
involvement  of  feelings  and  the  importance  of  experiences  which  comprise  the 
Global  Knowledge  necessary  for  specific  situations.  The  process  of  intuition  is  not  a 
linear  step-by-step  process  but  a  global,  non-linear  process  using  knowledge  and 
situation  cues  to  form  a  whole  perception  of  the  situation.  The  intuitive  process 
integrates  the  information  one  already  has,  the  new  associations  between  this 
information  which  provides  new  insights,  and  shreds  of  information  which 
formerly  had  no  meaning  to  develop  new  patterns  of  knowledge  and  new  solutions 
to  novel  problems. 

PROPERTY  -- 17  Hypnogogic  reverie  is  the  apparent  association  of  images  which 
appear  during  very  relaxed,  near  sleep  like  states.  We  are  not  concerned  with  these 
from  an  EDM  perspective. 

PROPERTY  —  19  The  environment  is  especially  important  to  the  intuitive  process 
in  three  ways.  First,  the  specific  training  and  scientific  culture  that  promote  and 
contribute  to  intuitive  thinking  are  necessary  for  the  pilot's  interpretation  of  the 
situation.  Second,  the  environment  should  be  relaxed,  familiar  and  free  from  stress. 
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Third,  this  relaxed  environment  can  spark  the  Eureka  experience  resulting  from  the 
unique  combination  of  circumstances  and  experience. 

PROPERTY  --  20  Transposition  does  not  necessarily  indicate  the  presence  of 
intuition  or  insight.  Insight  is  not  only  perception  of  relations,  but  also  awareness 
of  relations.  It  is  difficult  to  interpret  this  property  to  the  analysis  of  EDM. 

Now  that  we  have  reviewed  the  important  characteristics  of  Intuition  (i.e.,  3,  5,  7,  10, 
15, 16, 18, 19)  as  they  relate  to  EDM,  the  real  question  is  how  past  experiences  are 
used  to  solve  present  problems.  The  theory  of  intuition  explains  that  past 
experiences  are  grouped  into  specific  schema.  These  schema  are  the  data  with  which 
pilots  approach  new  situations.  It  remains  the  problem  of  the  current  research  to 
define  how  past  experience  is  used  to  define  present  goals.  Cognitive  psychology 
research  has  shown  that  "learning  by  doing"  or  activity  based  learning  which 
associates  ideas  with  experiences  (which  can  later  be  recalled)  is  necessary  to  achieve 
these  goals.  In  addition,  the  dependence  of  intuition  on  knowledge  that  is  learned 
through  experiences  and  sensory  contact,  reinforces  the  need  for  training  that 
includes  pilot  activities  both  physical  and  mental. 

Once  the  intuitive  ties  have  been  established  between  the  elements  of  my  ne  v 
experience  and  the  pilot's  knowledge  base,  it  is  highly  probable  that  th.  :  eLar^n  tt  ->s 
between  some  of  the  elements  of  the  problem  in  the  current  situatio-.  wi . 
analogous  to  previous  situations.  Since  experience  is  not  based  or  isoi-ie..  _  «r  sr  s 
but  of  elements  related  in  some  fashion,  the  power  of  intuition  is  somew  s. 
guaranteed. 
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APPENDIX  C 


NON-AVIATION  COGNITIVE  TRAINING  ACTIVITIES 


Three  approaches  to  enhanced  cognitive  training  are  described  in  this  section.  They 
are  examples  of  non-aviation  research  that  may  have  applicability  to  aviation. 


COGNITIVE  TRAINING 
EXAMPLES 


1 .  KREMUN-BICETRE,  PAWS  -  BRAIN  GYM 


2.  TVPt  C  «  CHAOB  CHANG  CRB  OR  IXPKRT  PROBLEM 
BOLVERB 


3.  CENTER  FOR  OECIMON  RESEARCH  U.  OF  CHICAGO 
"THOUGHT  TRIAL!- 


Fitness  is  given  as  a  month 
Reported  successes  include: 


C.l  Brain  Gym 

In  France,  neurologist  and  psychologist  Monique  Le 
Poncin-Lafitte  has  developed  a  program  to  improve 
memory,  logic,  attention  span  and  organizational 
ability.  She  calls  her  program  "Brain  Fitness"  and 
has  tailored  exercises  to  improve  mental  agility  and 
diversity  of  thinking  (Le  Poncin,  1990).  Brain 
■long  course  at  Kremlin-Bicetre,  a  hospital  near  Paris. 


•  32.5%  improvement  in  mental  agility 

•  47.3%  improvement  in  visual/spatial  memory 

•  20.1%  improvement  in  verbal  memory 

The  month  of  brain  exercises  is  divided  into  three  sessions  per  week,  with  each 
session  lasting  no  more  than  15  minutes.  The  program  of  exercises  is  based  upon 
five  types  of  mental  activities. 

1.  Perceptual  Activity  --  The  goal  here  is  to  train  the  sensory  memory.  Senses  of 
sight,  hearing,  smell  and  touch  are  developed  for  speed  as  well  as  acuteness  and 
discrimination.  These  exercises  prepare  the  trainees  (or  pilots)  for  dealing  with  time 
pressured  situations  or  problems  which  they  may  have  never  encountered 
previously. 

2.  Visual/Spatial  Activity  --  The  training  here  deals  with  both  sight  and  space. 
Attention,  associations,  elaborations  and  expectations  are  dealt  with  from  an 
information  processing  perspective.  Situational  awareness  is  heightened  by 
combining  the  perceptions  of  location,  time,  distance,  interval  and  orientation. 


3.  Structuralization  Activity  --  The  training  in  this  activity  teaches  the  individual 
the  abilities  of  organization,  patterns  construction/recognition  and  forming  a 
coherent  whole  out  of  individual  elements  observed  and  fitted  together.  This 
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cognitive  training  is  directly  relevant  to  the  recall  and  reconstruction  capabilities  so 
critical  to  long  term  memory. 

4.  Logical  Activity  --  This  training  uses  arguments,  deduction  and  reasoning  to 
discover  the  meaning  and  coherence  of  specific  situations  or  concepts.  Coupling 
new  ideas  to  old,  developing  procedural  rules  and  association  principles  results  in 
superior  information  processing. 

5.  Verbal  Activity  —  These  exercises  require  precise  communication  to  define 
abstract  representations  of  people,  relationships  or  situations.  Visual/verbal, 
aural/verbal  and  conceptual  activities  are  included.  Understanding  and  vigilant 
awareness  activities  enhance  self-monitoring  and  strategic  decision  making  ability. 

C.2  Type  C  Chaos  Changers  or  Expert  Problem  Solvers 

Both  theoretical  research  in  balance  theory  (specifically  cognitive  dissonance  and 
consistence)  and  common  psychology  agree  on  one  specific  problem  solving  point. 
People,  even  pilots,  will  do  everything  they  can  to  solve  problems  and  restore  a 
sense  of  balance,  control  and  order.  Some  try  to  wish  the  problems  away.  Others 
deny  or  ignore  the  real  issues.  Many  people  try  inappropriate,  negative  behaviors 
hoping  to  mask  what  is  really  going  on.  Others  think  that  by  applying  a  tried  and 
true  solution  from  the  past,  will  make  the  current  problem  go  away. 

At  the  Center  for  Studies  in  Creativity  at  Buffalo  State  College  in  Buffalo,  New  York, 
Dr.  Roger  Firestein  acknowledges  that  the  "hot  topic  right  now  is  problem  solving". 
Dr.  Firestein  and  his  colleagues  are  researching  the  characteristics  of  expert  problem 
solvers.  They  have  documented  that  the  effective  problem  solver  knows  how  to 
identify  the  right  problem  and  implement  the  right  solution  at  the  right  time.  This 
has  also  been  termed  "knowing  when"  (Dreyfus  and  Dreyfus,  1986).  That  is,  the 

Adaptive  Expert  Pilot  perceives  the  necessity  to  alter  ingrained  procedures  based 
upon  the  parameters  and  dynamics  (or  cues  and  context)  of  the  problem  or  situation 
encountered.  It  is  believed  that  this  "KNOWING  WHEN"  (an  almost  direct 
perception  of  the  proper  course  of  action)  may  provide  the  key  to  the  next 
generation  of  ADM  training. 

Dr.  Firestein  has  trained  thousands  of  individuals  in  effective  problem  solving 
using  a  two  phase,  six  step  method.  They  employ  a  stepwise  problem  solving  model 
that  uses  forward  inferencing  in  the  information  processing  solution.  First,  is  the 
problem  identification  phase;  second,  is  the  solution  generation  phase. 

1.  The  first  step  in  the  training  process  is  choosing  to  become  involved  in  the 
finding  of  a  solution.  Firestein  calls  this  "mess  finding".  He  also  recognizes  it  as 
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intuitive  in  that  the  problem  may  not  even  be  well  identified,  yet  the  pilot  or 
problem  solver  knows  that  something  is  wrong  and  requires  attention. 

2.  Finding  all  the  information  possible  about  a  situation  is  the  second  step.  The 
Center  for  Creative  studies  defines  data,  which  is  different  from  information  as 
including  all  of  the  feelings,  impressions,  thoughts  and  subjective  elements  of  a 
situation. 

3.  The  third  step  in  the  problem  solving  training  is  to  identify  and  prioritize  the 
most  important  pieces  of  the  situation.  In  some  cases,  this  involves  defining  more 
than  a  single  problem  as  comprising  the  current  situation. 

4.  This  first  step  in  solution  development  involves  defining  as  many  ideas  as 
possible  to  respond  to  the  problem.  These  can  be  done  individually  or  as  part  of  the 
a  group's  response  to  a  problem,  e.g.,  a  crew  in  the  case  of  aviation. 

5.  Selecting  alternative  solutions  is  the  next  step  of  the  training  program.  Each 
alternative  should  be  compared  to  the  probability  of  success  based  upon  its  own 
potential  for  solution  and  ability  to  resolve  concerns. 

6.  The  final  step  involves  acting  on  the  solution.  That  is  developing  a  stepwise 
implementation  plan  to  change  the  situation  and  solve  the  problem. 

Although  this  sounds  very  structured  and  linear,  Firestien  has  looked  at  problem 
solving  groups  with  and  without  the  training  in  dynamic,  unfamiliar  situations. 

He  has  recorded  that  the  trained  groups  start  solving  the  problems  from  their  first 
intuitive  perception  of  something  being  "not  right.  That  is,  the  effective  problem 
solving  training  teaches  individuals  techniques  to  produce  controlled  outcomes, 
solutions  even  in  chaotic  situations. 

C.3  Thought  Trials 

The  University  of  Chicago's  Graduate  School  of  Business  has  a  Center  for  Decision 
Research.  Dr.  Robin  Hogarth  of  the  Center  has  been  working  on  the  science  of 
decision  making  and  problem  solving  issues  since  1980.  One  of  the  most  interesting 
and  productive  efforts  to  improve  decision  related  skills  has  been  the  use  of  a  series 
of  mental  manipulations  or  "thought  trials"  to  improve  association  and  retention 
capabilities. 

In  these  exercises,  a  series  of  conceptual  problems,  hypothetical  situations  or 
imaginary  experiences  are  presented  in  a  rapid,  timed  sequence.  Individuals  are 
required  to  try  as  many  solutions  as  possible,  as  rapidly  as  possible  and  until  a 
satisfactory  solution  is  obtained.  Dr.  Hogarth's  theory  is  that  creative  thinking  and 
problem  solving  involves  precisely  the  same  principles  as  trial-and-error  learning 
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(Hogarth,  1989).  The  thought  trial  process  is  based  upon  the  theory  that  this  "blind 
variation"  or  randomness  builds  the  individuals  ability  to  provide  foresight  in  new 
situations  by  providing  a  very  large  number  of  associative  problem  solving  paths. 
Furthermore,  the  paths  which  lead  to  correct  solutions  are  retained  as  strong 
connections  since  the  overt  behavior  of  "finding"  a  solution  is  accompanied  by  the 
feelings  of  accomplishment  and  discovery.  The  "something  clicked"  or  "eureka" 
experience  usually  marks  the  successful  termination  of  the  process. 

An  important  implication  of  this  research  is  that  chance  plays  a  crucial  role  in 
decision  making  and  creative  thinking.  Dr.  Hogarth  doesn't  see  anything  wrong 
with  this.  On  the  contrary,  this  observation  substantiates  the  fact  that  expert 
decision  making  and  problem  solving  abilities  are  not  "something  special"  that  only 
a  few  select  individuals  have  or  can  acquire.  The  fact  that  creative  problem  solving 
is  not  a  unique  capability  has  been  pointed  out  by  other  researchers  throughout  this 
report.  However,  the  Center's  research  goes  one  step  further  and  recognizes  that  the 
individuals  can  "learn"  to  be  more  capable.  Four  important  sources  in  developing 
creative  problem  solving  are  delineated: 

1.  People  vary  in  their  capability  to  produce  a  wide  range  and  large  number  of 
thought  trials.  That  is,  imagination  and  motivation  to  produce  thoughts  and 
practice  problem  solving  are  skills.  An  individuals  "investment"  in  thinking 
increases  the  probability  of  being  successful  in  finding  solutions. 

2.  Longer  experience  and  exposure  in  a  field  equips  people  with  the  capability  of 
better  problem  solving  skills.  The  wider  exposure  increases  the  range  of  possible 
thought  trials  they  can  produce. 

3.  Working  and  training  in  a  field  that  fosters  independent  thought  and  exploratory 
behavior  can  liberate  the  individual  mental  talents  for  generating  creative 
solutions. 

4.  People  differ  in  their  ability  to  seize  upon  solutions  that  are  appropriate.  That  is, 
it  is  not  sufficient  to  be  able  to  generate  a  large  variety  of  solutions.  The  individual 
must  be  able  to  decide  which  solution  should  be  implemented. 

Research  using  thought  trials  has  shown  that  all  of  these  individual  differences  can 
be  learned  to  varying  degrees  (Hogarth,  1989).  All  are  extremely  important  to  the 
expert  aviation  decision  maker.  However,  the  fourth  trait  is  especially  appropriate 
to  aviation  problem  solving.  Pilot's  must  learn  to  facilitate  the  "editing"  or 
selection  process  while  retaining  the  ability  for  wide  scanning  of  possible  solutions. 
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This  wide  scanning  has  also  shown  to  underlay  an  individuals  ability  to  generate 
solutions  to  new  situations.  Therefore,  by  engaging  in  large  numbers  of  thought 
trials,  the  creative  expert  increases  his  chances  of  solving  new,  previously 
unencountered  problems  when  they  occur. 

In  summary,  the  thought  trial  research  has  shown  that  the  invention  of  creative 
solutions  depends  upon  generating  many  thought  experiments  or  combinations  of 
factors  capable  of  leading  to  solutions.  This  practice  plus  the  development  of  the 
mechanism  to  select,  test  and  retain  "successful"  solutions  may  be  precisely  what 
expert  pilot  decision  making  training  should  provide. 
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APPENDIX  D 


EDM  TRAINING  TOOL  EXAMPLES 


This  section  provides  three  examples  of  Expert  Decision  Making  tools  which  can  be 
used  for  either  training  decision  making  skills,  evaluating  cognitive  performance  or 
in  one  case  as  a  screening  tool  to  determine  a  priori  if  a  student  pilot  has  good 
decision  making  skills.  These  tools  were  selected  due  to  their  tailored  development 
in  the  aviation  human  error  reduction  field.  However,  they  are  not  intended  to 
represent  ihg  recommended  choices  or  even  a  prioritized  list.  Rather,  they  are  to 
represent  the  type  of  interactive  tools  which  can  expedite  and  enhance  how  pilot's 
think  and  solve  problems. 

D.l  COCKPIT  EMERGENCY  PROCEDURES  EXPERT  TRAINER  (CEPET) 

The  CEPET  system  was  introduced  in  Section  5.3.  It  is  a  PC  based  training  program 
for  helicopter  pilots  which  provides  self-paced  training  in  cockpit  procedures,  pilot 
decision  making  and  emergency  procedures.  The  program  has  several  modules 
with  flight  planning,  flight  scenarios,  situational  awareness  and  stress  management. 
The  pilot  can  practice  his  ability  to  recognize  problems  with  only  partial  cues  and 
choose  correct  solutions  as  discussed  previously  in  Section  5.3.  The  following 
paragraphs  provide  additional  detail  on  CEPET  capabilities. 

Other  CEPET  modules  allow  the  pilot  to  "fly"  a  particular  scenario  which  has  been 
associated  with  an  actual  accident,  incident  or  bad  decision  making  which  can  lead 
to  human  errors.  As  pointed  out  in  the  Hogarth  research,  practice  with  this  type  of 
decision  making  training  device  increases  knowledge  and  awareness  by  both 
providing  mere  associative  and  elaborative  connections  in  long  term  memory  and 
by  forcing  the  pilot  to  "select"  a  course  of  action.  This  interactive  device  can  also  be 
considered  to  provide  a  "synthetic  experience"  which  augments  actual  flight 
experience  or  simulator  training.  Exposure  to  a  large  number  of  problems  increases 
the  pilots  perception  and  recognition  skills  while  simultaneously  improving 
judgment  and  motivation  action. 

The  CEPET  type  of  training  is  also  available  on  interactive  video  systems  at  various 
flight  training  schools  such  as  Flight  Safety  International,  Simuflite,  etc.  These  types 
of  training  devices  are  strongly  recommended  for  EDM  training  since  they  help 
widen  the  pilots  exposure  to  problems,  recognize  promising  approaches,  deal  with 
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new  or  novel  failures  in  a  non-life  threatening  environment  and  teach  the 
important  EDM  skill  of  dealing  with  incomplete  or  ambiguous  data. 

D.2  Wondrous  Original  Method  for  Basic  Airmanship  Testing  (WOMBAT) 

Historically,  flight  personnel  selection  techniques  have  emphasized  skills  and  basic 
intelligence.  These  measures  are  reasonably  good  at  predicting  success  early  in 
flying  training  and  certainly  pilots  or  crews  without  reasonable  flying  skills  and 
intelligence  would  not  perform  ,vel‘  under  operational  stress.  Yet  these  pilots  do 
not  all  perform  the  same  ac  yrobier.  s  develop  later  in  the  actual  flight  situation 
Some  will  be  far  moro  sue.  es',- .  \'-.i  others  despite  their  apparent  uniformity  of 
skill.  In  fact,  large  d?  .erences  among  phots  are  discovered  as  nroblen  s  develop, 
situations  deteriorate,  te.vio.  mounts  3..0  confusion  results  from  actual 
operational  stress.  Research  (Roscoe  &  Ni  mtr,  1980)  into  the  underlying  reasons  for 
these  differences  has  sh:'---n  that  theie  a;  e  four  basic  characteristics  which  are  useful 
as  subjective  predictor;:  01  this  variability  in  pilot  behavior: 

1.  Attention  left  over  to  take  care  of  the  emergency  while  not  losing  control  of  the 
routine  operation  of  the  aircraft. 

2.  Ability  to  estimate  quickly  probable  outcomes  for  different  courses  of  action 

3.  A  sense  of  relations  which  allows  rapid  reordering  of  priorities  as  situatioas 
deteriorate  or  improve. 

4.  Decisiveness  of  action  in  the  face  of  indecision  by  others 

WOMBAT  is  a  pilot  decision  making  evaluation  and  selection  tool  designed  to 
screen  pilots  for  basic  abilities  or  attributes  credited  to  operationally  effective  pilots. 

A  microprocessor  based  system,  WOMBAT  integrates  generic  information 
processing  algorithms  with  new  pictorial  codes  and  new  decisional  paradigms  into 
several  interactive  subtasks.  The  thesis  behind  WOMBAT  is  that  today's  pilot  is  an 
information  manager  as  much  as  he  is  a  skilled  aviator.  Pilots  must  search  for, 
evaluate,  and  integrate  information  not  only  about  the  status  of  the  various  aircraft 
subsystems,  but  also  about  the  multiple  other  factors  associated  with  flying  the 
aircraft,  navigating,  communicating,  etc.  Performing  these  multiple  functions  is  the 
essence  of  airmanship,  but  the  human  characteristics  which  make  the  whole  process 
possible  can  be  measured  by  creating  complex  physical  problems  involving  multiple 
information  channels  with  pictorial  information  codes  which  compete  for  the 
pilot's  limited  pool  of  attention. 
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Therefore,  throughout  the  WOMBAT  testing,  the  pilot  is  presented  a  set  of  vertical 
scale,  dual  indicators  showing  the  relative  weightings  of  the  various  tasks  including 
current  performance  for  those  "in-play",  and  available  for  future  and  use.  The  sum 
of  the  products  of  the  weights  and  the  performance  difficulty  on  the  tasks  in-play 
minus  "investments"  in  information  and  "penalties"  for  mistakes  gives  the  pilot  an 
overall  indication  of  his  current  performance.  A  "predictor"  of  total  points  now  and 
a  final  ranking  based  upon  extrapolation  of  current  performance  for  the  time 
remaining  allows  the  pilot  to  decide  whether  or  not  to  try  different  tasks. 

Pilot's  using  WOMBAT  are  presented  primary  tasks  which  include:  single-axis  left 
hand  "altitude"  tracking  and  dual-axis  right  hand  "flight  path"  tracking.  These 
primary  tasks  serve  as  background  workload  for  all  other  time-shared  problem 
solving.  However,  each  tracking  task  has  an  autopilot  mode  that  can  be  engaged,  at 
a  cost  in  points,  to  free  the  pilot  to  pursue  other  methods  of  earning  points.  Of 
course  the  autopilots  are  prone  to  failures  that  vary  from  gradual  to  abrupt. 
Therefore,  these  autopilots  must  be  monitored  frequently  even  though  the  pilot  is 
working  on  other  tasks.  In  addition,  the  autopilots  are  "cheap"  in  that  they  provide 
less  accuracy  than  manual  tracking.  In  this  way,  the  pilot  must  face  the  decision  to 
revert  to  manual  tracking  when  the  problem  includes  high  weights  to  tracking 
accuracy. 

Although  the  tracking  tasks  are  primary  in  the  sense  that  they  cannot  be  ignored 
without  significant  penalty,  the  pilot  is  given  "secondary"  tasks  which  provide 
problems  and  opportunities  for  the  pilot  to  demonstrate  future  airmanship  skills. 
These  other  sources  of  activity  are  of  two  types.  One  type  provides  pieces  of 
information,  some  immediately  useful  (such  as  an  autopilot  failure  classification), 
some  useful  in  the  course  of  the  problem,  and  some  irrelevant.  The  other  type  will 
be  the  source  of  side  tasks,  the  performance  of  which  can  yield  rewards  (and 
penalties)  in  various  forms.  Problems  can  be  requested  and  rejected  at  no  cost  or 
accepted  for  scoring.  Both  of  these  types  of  activities  will  have  dual-aspect  indicators 
as  with  the  tracking  tasks. 

In  using  WOMBAT,  the  types  of  behavior  which  produce  high  scores  include: 
discovering  rules  that  may  not  be  explicit,  through  induction  and  deduction;  finding 
out  what  is  important  both  at  the  present  and  in  the  long  perspective  of  the 
situation,  and  allocating  priorities  accordingly;  perceiving  a  situation's  true 
characteristics  by  avoiding  subjective  biases  and  being  vigilant;  recognizing 
serendipitous  opportunities  quickly  and  seizing  them  before  the  opportunities  pass; 
remembering  to  do  a  specific  action  at  a  specified  future  time;  ignoring  irrelevant 
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distractions;  and,  finally,  tolerating  frustration  when  the  problem  intensifies  or  the 
situation  is  deteriorating. 

The  task  structure  outlined  places  a  high  premium  on  rational  attention  allocation 
strategy,  vigilance,  perception,  the  pilot's  ability  to  assess  what  is  really  important, 
and  the  ability  to  remember  to  do  things  that  are  important  in  a  specified  sequence. 
All  of  these  variables  have  been  shown  to  be  critical  to  the  pilot's  airmanship.  The 
design  of  the  testing,  scoring  and  tasking  is  meant  to  identify  those  pilots  who  will 
fail  in  an  emergency  to  perceive  the  true  level  of  risk  of  the  situation,  will  fail  to 
prioritize  tasks  correctly,  and  will  ignore  opportunities  to  seize  any  fleeting  choices 
or  decisions  to  avert  disaster. 

D.3  Cognitive  Effectiveness  Markers 

The  FAA/NASA/University  of  Texas  Crew  Performance  Project  has  been 
developing  evaluation  criteria  for  multi-crew  decision  making  and  problem 
solving.  In  the  air  line  environment,  the  1990s  have  become  a  period  of  focusing  on 
crew  resource  management  (CRM).  That  is,  a  time  when  the  true  team  resources 
are  described  not  only  in  terms  of  the  cockpit  crew  but  include:  the  cabin  crew, 
dispatch  and  maintenance  personnel.  This  team  performance  has  been  recognized 
as  a  critical  part  of  insuring  safe  and  efficient  flight. 

Crew  Resource  Management  (CRM)  training  in  this  context  has  been  developed  by 
several  air  carriers.  The  goal  of  this  training  is  to  improve  the  effectiveness  of  these 
groups  themselves  and  the  quality  of  interactions  between  them  and  the  cockpit 
crews.  However,  with  the  large  time  and  financial  investment  required  by  the 
airlines  to  support  these  programs,  it  has  become  critical  to  develop  evaluation  and 
assessment  techniques  both  as  a  means  of  tracking  the  effectiveness  of  the  training 
and  as  a  means  of  insuring  continuous  improvements. 

For  that  reason,  the  cognitive  and  interpersonal  markers  are  repeated  here  as  a  third 
example  of  the  types  of  tools  available  for  EDM  training  program  development  and 
as  guidelines. 

1.  COMMUNICATIONS  PROCESS  AND  DECISION  BEHAVIOR  SKILLS/ ATTITUDES 

Briefings/Debriefings 
Inquiry/Assertion 
Crew  Self  Critique 
Conflict  Resolution 
Communication 
Decision  Making 
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2.  TEAM  BUILDING  AND  MAINTENANCE  SKILLS/ATTITUDES 

Leadership 
Concern  for  Operation 
Interpersonal  Climate 
Group  Climate 
Automation  Management 

3.  WORKLOAD  MANAGEMENT  AND  SITUATIONAL  AWARENESS  SKILLS/ATTITUDES 

Preparation 

Planning 

Vigilance 

Workload  Distribution 
Distraction  Avoidance. 

The  actual  use  of  these  markers  in  pilot  flight  training  is  still  to  be  determined. 
However,  in  the  air  carrier  simulator  environment  the  subject  of  CRM  evaluation 
comprises  an  entire  report  itself  and  is  outside  the  scope  of  this  project.  The  reader 
is  referred  to  the  CRM  Industry  Workshop  report  (Lofaro,  1992,  and  the  SAE  ARD 
50037,  Adams,  Summwalt,  et  al)  for  additional  specifics  on  this  topic  including: 
philosophy,  scenario  development,  rating  criteria,  rating  forms,  etc. 
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APPENDIX  E 


EDM  TRAINING  METHODS 

Based  on  the  research  from  i  and  II  of  this  project  on  Expert  Decision  Making, 

three  promising  EDM  training/tejMng  methods  are  suggested  as  applicable  to  future 
pilot  training  in  decision  making.  These  are: 

•  Metacognition  Training  (under  evaluation  using  novice  pilots  at  Massey 

University  in  New  Zealand) 

•  Intuitive  Decision  Making  Testing  (evaluated  intuitive  decision  making  skills  of 

over  3000  managers) 

These  methods  have  been  empirically  tested  and  offer  encouraging  examples  of 
what  can  be  done.  The  FAA  is  encouraged  to  pursue  the  application  of  these 
methods  to  pilot  expert  decision  making  training. 

E.1  Metacognition  Training  Program 

The  School  of  Aviation  at  Massey  University  in  New  Zealand  has  been  developing 
and  evaluating  expert  decision  making  training  of  ab  initio  pilots  since  1990  (Hunt, 
1991).  Their  program  was  briefly  discussed  in  Section  5.3  additional  details  are 
provided  in  this  appendix.  The  training  program  at  Massey  focuses  on  the 
Approach  to  Learning,  the  pilot  Self-Concept  and  Metacognition.  The 
metacognition  training  teaches  information  processing  in  the  learning  context,  self 
management  of  the  situation,  and  memory  development  all  important  to  EDM. 

The  Massey  program  was  developed  as  a  result 
of  recognizing  that  human  error  accident 
causes  have  underlying  information 
processing  causal  factors  and  that  if  ab  initio 
and  novice  pilots  can  improve  the  efficiency 
and  effectiveness  of  their  "thinking",  and  thus 
improve  their  monitoring  and  evaluative  or 
problem  solving  functions,  then,  their  overall 
performance  as  pilot-in-command  will 
improve. 

It  was  with  these  considerations  in  mind  that  an  expert  decision  making  course  was 
developed.  The  course  sets  seven  objectives  for  the  student  pilots  which  are 
considered  to  be  evidence  of  improved  decision  making  (Hunt,  1991). 

1.  Increased  self  awareness  of  their  own  learning  processes. 

2.  A  more  cognitively  active  role  in  classroom  learning. 
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3.  More  control  over,  and  responsibility  for  their  own  learning. 

4.  Greater  ability  to  evaluate  and  monitor  their  own  learning  performance. 

5.  A  strategic  use  of  cognitive  (problem  solving)  strategies. 

6.  Improved  examination  performance. 

7.  A  deep,  holistic  approach  to  learning  rather  than  a  superficial  approach. 
Influences  on  Effective  Information  Processing 

Research  with  the  student  pilots  at  Massey  University  lead  to  the  conclusion  that 
there  was  a  complex  interaction  between  the  internal  and  external  factors  which 
influenced  a  pilot's  ability  to  confront  challenging  situations.  Internal  factors 
included  the  abilities  of  students  to  monitor  their  own  responses  to  situations 
(metacognition)  as  well  as  to  solve  the  basic  problems  presented.  These  situational 
awareness  functions  (monitor,  control  and  evaluate)  manifest  themselves  when 
pilots  experience  an  awareness  of  the  range  of  possible  decisions  and  are  able  to 
select  appropriate  solutions  for  a  specific  problem  or  task.  As  they  work  through  a 
problem,  the  metacognitive  student  will  regularly  check  their  progress.  When  a 
failure  in  the  solution  occurs,  they  identify  and  resolve  it  quickly. 

The  Course 

Due  to  the  dilemma  of  developing  a  course  for  the  average  student  while  trying  to 
teach  expert  decision  making,  it  was  necessary  to  develop  a  cor  iproi.  use  between 
individual  attention  and  group  learning.  Therefore,  the  course  begins  with  an 
overview  of  information  processing  and  the  factors  which  can  influence  effective 
learning.  The  basic  course  syllabus  provides  the  students  with  the  appropriat 
knowledge  and  procedures  to  interpret  and  understand  their  own  "thn.king"  or 
decision  making  processes. 

Very  early  in  the  course  the  pilots  are  introduced  to  time-pressured  and  dynamic 
decision  situations.  They  are  required  to  identify  actions,  set  priorities  and 

apportion  their  time  to  specific  tasks. 
They  are  encouraged  to  continually 
review  and  evaluate  their  decisions. 
The  following  rating  factors  are  used 
in  the  evaluation  of  the  student  pilot 
performance.  First,  they  are  rated  on 
their  ability  to  analyze  objectives  and 
evaluate  the  content  of  the  task  or 
situation.  Analyzing  objectives 


58 


according  to  Content  Types  and 
Processing  Level  provides  the  basis  for 
deciding  which  cognitive  strategies  to 
use  and  the  basis  for  evaluating  lh-_ 
completeness  of  the  information  about 
the  problem  or  situation. 

Second,  they  are  evaluated  on  the  appropriate  application  of  conceptual  knowledge 
(facts)  or  procedural  knowledge  (rules)  used  to  address  the  situation.  Identifying 
content  types  triggers  the  student  to  the  importance  of  the  available  structures  and 
further  expands  his  association  of  elements  of  the  problem.  This  also  provides  a 
check  to  the  student  that  all  available  structural  components  are  used  as  resources. 
Third,  they  are  evaluated  on  the  remembering  and  retention  phase.  For  each 
objective  identifying  the  required  processing  level  helps  the  student  to  choose  the 
appropriate  cognitive  strategies  for  learning. 

On  a  processing  level  basis,  the  lowest  level  of  learning  is  "remembering  verbatim". 
This  simply  requires  the  student  to  be  able  to  reproduce  information  exactly  as  it  was 
presented.  This  kind  of  remembering  task  yields  best  to  mnemonics  strategies  such 
as  acronyms,  chunking  and  sequential  identification.  Learning  tasks  that  require 
deeper  processing  rely  on  the  meaningfulness  of  the  information.  Massey  calls 
these  "remember  paraphrase"  and  "use".  This  type  of  learning  to  requires  the 
students  to  "comprehend"  or  make  sense  out  of  the  information  in  a  manner  that 
they  can  express  in  their  own  words.  It  also  requires  understanding  relationships 
between  facts,  objects,  results,  as  they  occur  in  context.  This  is  accomplished  through 
elaborations  and  associations  which  stimulate  prior  knowledge  and  establish  the 
necessary  connections  between  existing  structures  and  new  material. 

Once  information  is  taken  in,  understood  and  structured  for  use,  it  must  be 
"anchored"  in  memory  for  ease  of  retrieval.  Current  literature  on  expertise 
recognizes  the  ability  to  use  and  apply  knowledge  in  "real  situations"  is  as  much 
dependent  on  easy  retrieval  as  it  is  of  organization  and  storage.  The  Massey  course 
teaches  two  strategies  for  anchoring  knowledge:  Associating,  i.e.,  creating  analogies 
which  elaborate  on  the  knowledge  base;  and,  Generating,  i.e.,  practicing  applying  the 
information  to  new  situations.  Both  techniques  are  used  to  force  the  students  to  go 
further  than  just  resolving  the  current  situation.  These  traits  are  based  upon  the 
observation  that  expertise  comes  through  the  use  of  knowledge  and  not  just  by  the 
availability  of  facts. 
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Metacognition  or  Super  Situational  Awareness 


In  addition  to  teaching  expert  decision  making,  Massey  students  are  taught  to  be 
aware  of  their  own  problem  solving  abilities.  They  are  taught  to  assess  how  well  or 
how  poorly  they  are  performing  and  can  take  alternative  actions  to  improve  their 
performance.  These  self-monitoring  activities  are  another  typical  characteristic 
observed  for  experts  in  other  fields.  The  cognitive  psychology  field  refers  to  this 
activity  as  metacognition.  In  aviation  terms,  the  monitoring  and  continual 
evaluation  of  the  viability  of  a  flight  is  referred  to  as  situational  awareness.  Insofar 
as  the  Massey  students  exceed  the  normal  monitoring  of  flight  progress  and  the 
situation  and  actually  monitor  their  own  decision  making  and  problem  solving  this 
could  be  called  Super  Situational  Awareness. 

At  Massey,  raising  student  pilots  awareness  of  their  metacognitive  functioning 
begins  when  they  are  given  their  learning  profiles  at  the  start  of  the  course.  The 
students  are  each  asked  to  evaluate  the  "truth"  of  these  profiles  and  reflect  on  their 
own  learning  processes.  This  awareness  is  continued  by  the  students  in  a  journal  or 
diary  in  which  they  keep  records  of  their  learning  experiences,  successful  decisions, 
problematic  outcomes,  thoughts,  feelings  and  insights.  These  structured,  written 
exercises  discipline  the  pilots  into  a  regime  of  planning,  analyzing  and  monitoring 
their  own  cognitive  performance.  This  exercise  also  provides  the  students  with 
support  for  developing  metacognitive  skills  before  they  have  been  able  to 
completely  automate  or  internalize  these  strategies. 

Summary 

The  pilot  who  is  able  to  recognize  that  a  particular  decision  or  problem  solving 
strategy  isn't  working  and  can  select  another  strategy  from  a  repertoire  will  complete 
the  task  more  quickly  and  completely  that  one  who  does  not  identify  this  deficiency 
in  cognitive  processing  and  is  therefore  unable  to  take  remedial  action.  Since  a 
critical  ingredient  in  decision  making  is  access  to  relevant  information,  the  Massey 
training  seems  headed  in  the  direction  of  providing  the  required  expansion  in 
knowledge  base  and  ease  of  retrieval  in  addition  to  the  enhancement  in  self¬ 
monitoring.  Without  appropriate  knowledge,  pilots  cannot  make  good  decisions. 
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E.2  Intuitive  Decision  Making  Testing 

A  successful  field  testing  of  intuitive  decision  making  skills  of  over  3000  managers 
was  performed  throughout  the  U.  S.  in  a  wide  variety  of  organizational  settings  in 
the  1980s.  These  settings  included  business,  government,  education,  military  and 
health  and  a  wide  range  of  management  levels.  This  experience  and  the  positive 
results  suggest  a  cost-effective  opportunity  to  transfer  developments  in  appropriate 
testing  or  training  from  the  management  field  to  aviation.  The  following  details  are 
provided  as  an  expansion  of  the  summary  description  previously  introduced  in 
Section  5.4. 

Field  Testing 

The  groups  tested  included  a  wide  range  of  different  organizations  and  settings 
including  private  sector  chief  executive  officers,  emergency  preparedness  military 
personnel,  community  college  presidents,  state  health  and  rehabilitative  services 
managers,  city  managers,  state  legislators  and  staff,  professional  civil  servants,  and 
executives  across  the  country.  Over  the  series  of  tests,  two  different  but 
complimentary  test  instruments  were  used.  The  instrument  used  to  measure 
underlying  intuitive  ability  consisted  of  twelve  questions  selected  from  the  Myers- 
Briggs  Type  Indicator  (MBTI),  a  psychological  instrument  that,  among  other  things 
measures  a  person's  ability  to  use  intuition,  as  contrasted  to  deductive  reasoning,  to 
make  decisions  (Myers,  1987).  This  instrument  was  selected  for  use  because  it  has 
been  widely  applied  in  practical  settings  across  the  country,  and  has  been  proven  to 
have  a  high  degree  of  reliability  and  validity  as  a  test  instrument.  A  scale  was 
constructed  for  this  part  of  the  test  so  that  each  manager  could  be  ranked  exactly 
from  top  to  bottom  on  how  they  scored  individually,  and  also  how  they  compared 
to  other  managers  taking  the  test.  The  maximum  score  was  12  (highly  intuitive) 
with  a  minimum  score  of  0  (little  or  no  intuitive  ability  apparent). 

The  second  instrument  measured  whether  the  managers  tested  actually  used  their 
intuitive  ability  to  make  important  decisions,  and  if  so,  how.  These  questions  were 

designed  for  follow-up  testing  based 
on  the  pattern  of  responses  received 
after  administering  the  first  test. 

The  test  was  also  designed  to  obtain 
data  on  a  number  of  related 
questions  such  as  whether 
executives  who  used  intuition  to 
guide  their  decisions  actually  shared 
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this  fact  openly  with  their  colleagues  or  kept  it  a  secret.  When  used  together,  these 
two  tests  became  powerful  tools  for  measuring  a  manager's  intuitive  decision 
making  ability.  They  may  offer  interesting  parallels  suitable  for  use  throughout 
aviation. 

AIM  Results 

Responses  were  stratified  for  such  key  variables  as  level  of  management,  level  of 
government,  sex,  occupational  specialization,  and  ethnic  background.  All  of  the 

responses  were  analyzed  by  computer 
and  all  the  findings  were  subject  to 
statistical  significance  tests.  The  findings 
from  this  national  testing  dramatically 
indicated  that  intuitive  management 
ability  appears  to  vary  by  management 
level,  by  level  of  government  service,  by 
sex,  by  occupational  specialty  and,  to 
some  degree,  by  ethnic  background 
(Agor,  1986).  Intuition  appears  to  be  a  skill  that  is  more  prevalent  as  one  moves  up 
the  management  (experience)  ladder.  Sample  results  are  shown  on  the  left. 

Top  managers  in  every  sample  group  tested  scored  higher  than  middle/lower  level 
managers  in  their  underlying  ability  to  use  intuition  to  make  decisions.  It  also 
appears  that  the  higher  one  goes  in  the  level  of  government  service  (from  county  to 
national),  the  greater  the  ability  to  use  intuition  becomes.  Finally,  it  appears  that  top 
managers  rely  most  frequently  on  their  intuition  since  they  are  constantly  called 
upon  to  make  choices  in  turbulent  environments  where  problems  do  not  lend 
themselves  to  serial,  deductive  or  probabilistic  reasoning. 

Because  of  these  strong  parallels,  and  the  documented,  national  findings  across  a 
wide  range  of  fields  that  the  AIM  approach  to  testing  is  reasonable,  a  similar 
approach  suitable  for  testing  the  pilot  memory,  information  processing  and  problem 
solving  skills  identified  as  important  to  EDM  was  proposed  in  Section  5.4.  As  with 
the  AIM,  the  proposed  self-assessment  test  will  be  composed  of  two  parts.  Each  part 
will  contain  12  questions  or  problems  which  will  be  administered  as  independent 
self-assessment  tests  at  two  different  time  periods  to  reduce  the  immediate  influence 
of  Fart  I  on  Part  It.  The  self-test  will  provide  each  pilot  with  a  rating  matrix  and 
personnel  decision  making  style  profile  similar  to  the  "Hazardous  Attitude"  profile 
used  successfully  in  the  basic  ADM  training.  Part  I  will  be  MBTI  based  and 
determine  the  pilots  ability  to  use,  and  tendency  to  prefer,  intuitive  decision 


making.  Part  II  will  be  cognitive  skill  based  and  will  evaluate  pilots  mental 
attributes  which  contribute  to  his  actual  use  of  various  cognitive  styles  for  decision 
making  and  problem  solving,  up  to  and  including  intuition. 
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APPENDIX  F 


EDM  TRAINING  ASSESSMENT 


Experienced  pilots  can  be  viewed  as  experts  on  how  to  safely  fly  aircraft  and  how  to 
safely  deal  with  the  full  range  of  emergency  situations  that  can  occur  while  flying. 
The  design  of  testing  methods  for  this  type  of  expertise  has  two  goals.  First,  the  test 
should  be  an  assessment  of  a  given  pilot's  ability  to  cope  with  emergency  situations 
and  provide  descriptive  feedback  on  a  given  pilot's  mental  processes  during  this 
type  of  decision  making  and  problem  solving.  Second,  taking  the  test  should  be  a 
valuable  learning  experience  and  the  test  should  provide  explicit  feedback  on  the 
information  that  needs  to  be  considered  for  the  presented  situation  and  detailed 
discussion  on  different  strategies  to  deal  with  that  situation. 

In  addition,  in  comparing  the  thoughts  generated  by  the  pilots  who  are  taking  the 
test  to  the  detailed  feedback  about  different  strategies  from  the  second  goal,  the  pilots 
should  be  stimulated  to  consider  why  they  didn't  consider  various  options  or 
selected  sub  optimal  strategies.  Tests  designed  in  this  manner  would  provide  an 
essential  training  tool  to  attain  continued  training  and  enhancement  in  pilot 
decision  making  whereas  emergency  situations  are  infrequently  encountered  in  real 
life.  The  basic  design  of  the  type  of  tests  required  for  developing  adaptive  and 
intuitive  expert  pilot  decision  making  draws  heavily  on  the  material  previously 
presented  throughout  this  report.  However,  for  clarity  and  ease  of  reference,  the 
next  section  will  briefly  outline  the  general  insights  regarding  EDM  and  IDM 
processes  and  findings  relevant  to  the  assessment  of  testing  and  training  methods. 

F.l  Methodology 

The  central  problem  in  studying  expert  decision  making  in  aviation  is  common  to 
all  research  on  expertise  in  that  the  process  to  be  studied  occurs  in  complex  and 
highly  interactive  situations.  Medical  experts  diagnosing  the  disease  of  a  patient 
engage  in  extended  interviews,  medical  tests  and  physical  exams.  Expert  athletes  are 
exposed  to  dynamically  changing  situations  throughout  their  competitive  games. 
Similarly,  pilots  are  exposed  to  continuously  changing  environments  in  which  their 
responses  and  actions  will  interactively  change  the  situation  at  hand.  Hence,  a 
given  expert  will  most  likely  never  encounter  the  identical  situation  twice  in  a  lii  ■ 
time,  nor  will  two  experts  ever  encounter  the  identical  situation  or  problem 
environment.  This  constraint  of  not  being  able  to  observe  many  experts  (as  well 
novices)  in  the  same  situation  is  a  major  challenge  for  research  on  pilot  expertiS' 

The  general  solution  to  this  problem  is  to  analyze  carefully  the  real  life  expert^..  to 
identify  brief  segments  of  the  expert's  behavior  that  corresponds  to  a  natural 
occurring  task.  This  type  of  job  task  analysis  was  suggested  as  the  next  significant 
step  in  the  development  of  decision  making  training  by  the  participants  at  the  May 
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1992  workshop  (Adams,  &  Adams,  1992).  An  example  of  the  type  of  cognitive  task 
analysis  (CTA)  was  presented  by  Kevin  Smith  from  United  Airlines.  His  detailed 
task  analysis  of  taking  off  with  and  engine  failure  at  VI  provided  12  decision  points 

which  required  a  specific  response. 

By  examining  each  decision  point,  it  is 
possible  to  record  the  expert  decision 
vs.  the  novice  decision  based  upon  a 
consensus  of  high  time,  accident  free 
pilots.  This  situation  or  decision  task 
could  then  be  presented  in  isolation  to 
a  large  number  of  other  pilots  and  ask 
them  to  select  the  best  possible 
solution. 


In  many  task  analyses  it  is  possible  to  explicate  a  limited  number  of  sequences  of 
thoughts  that  will  adequately  describe  how  subjects  generate  the  best  answer  or 
some  alternative  answer  for  a  given  task.  These  alternative  thought  sequences 
make  it  feasible  not  only  to  describe  other  subjects'  efforts  to  complete  the  task,  but 
would  provide  the  potential  of  giving  feedback  to  subjects  generating  sub-optimal 
answers/ actions.  Analyses  of  expert  performance  using  this  type  of  task  analysis  has 
revealed  a  perspective  of  general  characteristics  of  experts’  cognitive  processes. 

F.2  Desired  Characteristics  of  EDM  Processes 


Based  on  the  research  on  expert  decision  making  from  Sections  4.0  and  Appendix  A, 
a  consistent  picture  of  the  organization  of  expert  knowledge  and  problem  solving 
has  emerged  across  a  wide  range  of  fields.  In  a  task  analysis  of  a  medical  diagnosis, 
for  example,  all  possible  diagnoses  can  be  sequentially  checked  to  see  if  they  can  be 
ruled  out.  If  only  a  single  possible  diagnosis  remains,  the  diagnosis  is  complete. 

This  is  a  useful  method  to  analyze  the  task  and  how  presented  information  can  be 
used  along  with  other  knowledge  to  eliminate  alternative  diagnosis.  However, 
human  medical  experts  do  not  proceed  in  this  fashion.  The  medical  expert 
sequentially  integrates  the  presented  information  and  will  only  explicitly  consider  a 
very  small  set  of  reasonable  diagnosis.  In  dynamically  changing  environments,  like 
aviation,  the  expert  maintains  an  updated  representation  of  the  current  situation  or 
in  medicine,  the  proper  diagnosis. 

For  the  type  of  tasks  presented  to  pilots,  from  setting  up  a  procedure  turn  for  an 
instrument  approach  to  dealing  with  engine  failures,  the  first  step  in  expert 
performance  is  to  rapidly  encode  the  presented  information  to  generate  an 
integrated  representation  of  all  relevant  information  about  the  current  situation  or 
task.  Once  this  integration  is  complete,  the  expert  virtually  automatically  retrieves  a 
small  number  of  appropriate  actions/solutions.  This  retrieval  is  not  a  deliberate 
search  for  alternatives,  but  the  retrieved  potential  actions  simply  occur  to  the  expert 
due  to  a  direct  recall  based  upon  elaborate  internal  knowledge  representations. 
Subjectively,  this  direct  retrieval  process  is  perceived  as  intuition.  At  this  first  phase 
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the  expert  has  no  assurance  that  these  actions  are  the  best  nor  that  they  completely 
reflect  all  relevant  constraints  of  the  situation.  It  is  during  the  following  second 
phase  that  these  retrieval  alternatives  are  carefully  evaluated  and  in  some  cases 
rejected  in  favor  of  better  alternatives. 

For  highly  familiar  situations  previous  experience  and  earlier  corrected  mistakes 
essentially  assure  that  appropriate  and  correct  actions  are  retried.  However,  for 
unfamiliar  situations  the  retrieved  actions  can  be  viewed  as  heuristics  and  thus  are 
not  necessarily  best  or  even  entirely  appropriate.  Research  of  expert's  decisions  in 
other  fields  has  shown  that  their  intuitive  ideas  are  good,  but  that  quite  often  (30- 
50%)  of  the  time,  they  are  rejected  after  a  more  careful  evaluation  of  the  possible 
alternatives  for  the  situation  presented  (de  Groot,  1978). 

A  large  body  of  research  has  studied  the  integrated  representation  of  situations 
encoded  by  experts  in  expanded  memory.  Novices  and  lesser  experts  are  simply 
unable  to  internally  represent  the  presented  information  and  memory  studies  show 
that  these  subjects  can  only  recall  a  fraction  of  the  presented  information  after  brief  5 
second  exposures.  In  contrast,  experts  can  virtually  recall  all  relevant  information 
perfectly.  The  experts’  integrated  representation  of  the  current  situation  plays  a 
critical  role  in  the  experts'  more  systematic  evaluation  of  alternatives  during  the 
second  evaluative  phase.  The  expert  can  plan  out  consequences  of  various  actions 
and  internally  represent  and  evaluate  these  future  states. 

The  amount  of  time  available  for  this  second  phase  of  systematic  evaluation  differs 
from  situation  to  situation  and  from  time  to  time  in  the  same  situation.  There  are 
several  methods  to  anticipate  issues  and  perform  necessary  evaluation  that  will 
allow  an  expert  under  real-time  constraints  to  reduce  the  necessary  processing  at  the 
point  of  making  the  decision. 

1.  Planning  and  preparation  of  alternatives  before  the  situation  or  procedure  is 
required  in  real  flight  situations. 

2.  Concurrent  updating  and  evaluating  of  information  at  the  first  possible 
opportunity. 

In  virtually  all  areas  of  expertise,  the  most  effective  learning  takes  place  after  the 
event,  when  the  course  of  events  can  be  carefully  evaluated  without  any  real-time 
constraints.  Analyzing  videotapes  of  LOFT  simulator  sessions  is  a  good  example. 
Most  of  the  actual  learning  and  further  improvement  of  the  expert  performance  is 
due  to  training  inspired  by  such  post-event  analyses.  In  addition,  descriptions  and 
recordings  of  other  pilot’s  and  crew's  performance  can  similarly  be  used  as 
opportunities  for  learning.  Accident/incident  videotape  re-enactments  are  an 
example  here. 
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F.3  Implications  for  EDM  pilot  testing/training 

The  preceding  discussion  of  research  results,  principles  of  experts  and  characteristics 
of  expert  problem  solving  produce  a  general  format  for  the  desired  pilot  testing  of 
expert  decision  making.  The  tests  should  be  composed  of  a  series  of  items.  Each 
item  has  three  major  parts  and  some  variations. 


PART  I:  A  detailed 
description  of  a  flight  situation 
(or  pre-flight) 

•  a  picture  of  the  instrument 
panel  with  appropriate 
situation  readings 

•  a  picture  of  external 
weather,  traffic,  runway 
configuration,  etc. 

PART  la:  On  the  following 
page  a  list  of  questions  testing 
whether  the  pilot  can  recall 
most  of  the  relevant 
information  presented  on  the 
first  page 

TEST  FOR 

GENERATED 
INTEGRATED 
REPRESENTATION  OF 
THE  CURRENT 
SITUATION 

PART  II:  On  a  separate  page 
present  a  description  of  a 
critical  event  and  a  request  for 
a  decision  about  relevant 
actions 

PART  Ha:  As  an  optional 
exercise,  present  (on  a  separate 
page)  a  listing  of  3-5  possible 
courses  of  action  and  a  request 
to  select  one  of  them  as  the 
one  closest  to  their 
spontaneous  answer 

TEST  FOR  HOW 

MUCH  TIME  IS 
REQUIRED  USING  A 
STOPWATCH  OR  AN 
EGG  CLOCK  RINGING 
AT  SOME  PRESET 

TIME 

PART  III:  Present  an  expert 
analysis  of  the  situation 
outlining  possible  courses  of 
action  and  discussion  of  the 
possible  consequences 
associated  with  each  choice 

PART  Ilia:  As  an  alternative 
to  PART  III,  present  the  "best" 
course  of  action  (or  best 
courses  when  two  are  equally 
acceptable).  Discuss  sub- 
optimal  or  incorrect  actions 
sequentially  providing 
detailed  discussion  of  their 
unfavorable  consequences  and 
plausible  reasons  for  their 
selection 

TEST  FOR  DISREGARD 
OR  NEGLECT  OF 
CRITICAL  PIECES  OF 
PRESENTED 
INFORMATION 

UNDER  PART  I 

The  following  considerations  are  critical  to  the  scenario  generation: 


1.  Selecting  scenarios  with  realistic  situations  and  critical  events. 

2.  Adapting  scenarios  to  make  a  task  analysis  with  a  small  number  of  reasonable 
sequences  of  actions  possible. 


3.  Evaluation  of  test  performance  and  self-evaluation  of  employed  decision  making 
processes. 
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F.4  Selecting  Realistic  Scenarios  with  Critical  Events 

It  would  be  preferable  to  elicit  real-life  events  from  experienced  pilots  and  then  try 
to  adapt  them  to  the  recommended  three-part  format.  Being  able  to  present  the 
encountered  scenarios  as  events  from  real  experiences  will  improve  the  face  validity 
of  the  test.  This  development  should  also  strive  for  scenarios  and  critical  events 
that  require  .  imilar  actions  for  large  classes  of  airplanes  or  could  be  easily  adapted  to 
a  specific  airplane  type. 

Another  source  of  scenarios  and  critical  events  can  be  abstracted  from  simulator 
training  programs,  which  are  no  longer  in  use.  A  final  source  would  be  to  seek  out 
clear  critical  events  and  then  adapt  them  with  a  typical  scenario.  At  least  some 
small  portion  of  the  problems  should  be  very  different  involving  combinations  of 
error  (a  hidden  error  in  the  current  situation  plus  a  critical  event).  The  pilots 
should  be  told  about  the  range  of  possibilities  prior  to  the  start  of  taking  of  the  test. 

F.5  Adaptation  of  Scenarios:  Task  Analysis 

The  real-life  scenarios  will  require  substantial  adaptation  to  yield  a  small  number  of 
plausible  courses  of  action.  Potential  courses  of  action  can  be  eliminated  by  adding 
information  in  the  scenario  (PART  I).  Simple  courses  of  action  can  be  made  more 
difficult  by  manipulating  sets  (e.g.,  information  about  an  earlier  problem  with  the 
hydraulics  system)  and  inclusion  of  other  irrelevant  information  on  PART  I. 

Another  less  theoretical  approach  is  to  plan  a  relatively  large  field  testing  with 
scenarios  without  explicitly  given  options  and  then  categorize  the  volunteered 
options  into  groups  that  can  be  restated  as  actual  options  for  the  real  test.  As  a  result 
of  this  field  testing,  information  can  be  added  and  taken  away  to  adjust  the  difficulty 
of  each  item. 

During  this  phase  it  is  essential  to  have  access  to  a  group  of  super-expert  pilots  to 
assure  realism  and  to  write  the  feedback  portion  (PART  III  and  PART  Ilia). 

F.6  Test  Performance  and  Self-Evaluation  of  the  Decision  Making  Processes 

Using  the  proposed  design  of  items  with  a  fixed  set  of  alternatives  and  only  one  or 

two  alternatives  being  the  best,  scoring  of  the  test  should  be  straightforward.  It  may 

also  be  possible  to  categorize  sub-optimal  or  incorrect  alternatives  in  different  , 

categories  such  as: 

a.  forgot  or  neglected  facts  given  in  the  situation  (  PART  I)  • 

b.  Violated  FARs  or  Company  procedures  (SOPs) 

c.  Insufficient  knowledge  about  the  aircraft,  systems,  procedures,  etc. 

d.  Willingness  to  accept  a  higher  than  necessary  risk 
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It  may  be  difficult  to  construct  alternatives  that  uniquely  reflect  one  of  these  error 
sources,  so  the  pilot  may  have  to  check  off  one  or  more  of  these  when  confronted 
with  the  feedback  in  PART  Ilia.  These  errors  could  then  be  summed  to  provide  an 
expertise  profile. 

There  are  a  number  of  possibilities  for  more  detailed  self  assessments  once  a 
preliminary  version  of  the  test  is  developed.  At  that  point,  a  sample  test  of  pilots 
should  be  given  a  "rhink-aloud"  type  of  test  and  retrospective  reports  on  their 
thought  processes  recorded.  From  these  sample  decision  making  "records"  there  are 
other  possible  directions  that  the  testing/training  could  proceed: 

1.  Assessment  of  the  pilots'  memory  for  the  presented  information  in  PART  I  after 
a  standardized  exposure  time  evaluating  the  following: 

a.  Recall 

b.  Test  of  meaningful  integration 

c.  Test  for  ability  to  generate  likely  problems  or  incidents  given  a  presented 

current  situation  (Comprehension  and  Inference) 

d.  Questions  about  what  kind  of  preparation  that  should  have  been  already 
done  for  a  flight  like  the  one  described  in  PART  I  (Comprehension  and 
Interference) 

2.  Assessment  of  the  pilots'  ability  to  respond  rapidly  and  access  intuitive  courses  of 
action.  By  varying  the  time  to  come  up  with  an  option  from  2-5  seconds  (including 
the  reading  of  the  situation  description  and/or  the  critical  incident)  to  30-45  seconds, 
one  should  be  able  to  vary  the  reliance  on  pure  intuition  and  intuition  with 

It  is  possible  that  pilots  differ  and  that  some 
always  are  satisfied  with  intuition,  i.e.,  no 
systematic  evaluation  even  with  long  times, 
others  may  simply  be  unable  to  rapidly  respond 
and  require  generation  of  alternatives,  which 
will  incorporate  some  degree  of  evaluation. 
Roughly  four  groups  could  be  postulated  as 
shown  on  the  left. 


Further  detailed  development  of  scenarios  and 
discussion  of  self-validation  will  require 
feedback  from  the  readers  of  this  report,  the 
critical  review  of  the  FAA  Technical  Monitor,  other  appropriate  FAA  personnel  and 
others  pilots  with  varying  amounts  of  experience. 


subsequent  systematic  evaluation. 


EDM  ASSESSMENT  GROUP 


COGNITIVE 

ASSOCIATIVE 

AUTONOMOUS 

INTUTIVE 

_ 

